1 Release 1.8.11 on 08/01/2014
3 * Expose http decoder parameters as config
4 * Client Parallel request Exceptions are not reported to failure detector
5 * ClientRequest Executor has infinite timeout and log cleanup
6 * Threshold failure detector bug fixes
8 Release 1.8.10 on 07/23/2014
10 * Registering slops for parallel puts that fail on QuotaExceededException
12 Release 1.8.9 on 07/02/2014
14 * Adding backwards compatibility checks in AdminClient
16 Release 1.8.8 on 06/24/2014
18 * Wrap mime multipart into a mime message to ensure headers are updated in
20 * Fix transport client instantiation and HttpClientFactory shutdown
21 * Fix build.gradle for setting java builder property in .project file
23 Release 1.8.6 on 06/11/2014
25 * Fix related to Quota metrics (store level instead of aggregate)
27 Release 1.8.6 on 06/11/2014
29 * Fix related to quotas - checking per store instead of aggregate
31 Release 1.8.5 on 06/06/2014
32 *Gradle related changes
34 - empty voldemort-contrib directory
35 - gradle wrapper to fetch required version
37 Release 1.8.4 on 06/03/2014
39 * Fix RESTClientFactory for a clean shutdown
40 * Add check for SocketAndStreams in sendAndReceive
41 * Added the ability to query a single store using AdminClient / admin tool
43 Release 1.8.3 on 05/27/2014
45 * Add debug log messages for CoordinatorService
46 * Fix NetworkClassLoader bug when running in Windows OS (Carlos Tasada)
47 * Add junit support for gradle
48 * Fix bind failures while running tests
49 * Add overwrite option to fork lift tool
50 * Use standard logger in the StreamingClient constructor
51 * [python client] Fixed reconnect in _execute_request (Søren Holbech)
52 * Fix the admin metadata check command and did same change for old admin command.
54 Release 1.8.2 on 05/09/2014
56 * Bug fix in MetadataStore regarding updation of routing strategy
58 Release 1.8.1 on 05/01/2014
60 * Fix AbstractStoreClientFactory for backwards compatibility
61 * Cleaned up logging messages
62 * Unit tests for big Binary data
63 * Refactored voldemort admin tool - Phase 1 and Phase 2
65 Release 1.8.0 on 04/21/2014
67 * Split the stores.xml into individual stores on Voldemort server
68 * Maintaing backwards compatibility of MetadataStore
69 * DefaultStoreClient bootstraps using store name (instead of stores.xml)
70 * New Admin tool functionality to update stores (instead of replace)
72 Release 1.7.3 on 04/16/2014
74 * Fix RESTClientFactory to return a LazyStoreClient
76 Release 1.7.2 on 04/15/2014
78 * Quota limiting related fix
79 * Coordinator related fix
81 Release 1.7.1 on 04/10/2014
85 Release 1.7.0 on 04/09/2014
87 * Improves BDB coversion process and documentations
88 * Adds scala build support and shell (Saurabh Bhatia)
89 * Add removeStorageEngine call to StorageConfiguration
91 Release 1.6.9 on 03/19/2014
93 * Fix a big where StreamingClient will throw NPE if not initialized
94 * Fix multiple broken tests
95 * Modified RO Rebalance Test to cover one more case
96 * Adding thin client shell for Voldemort Coordinator
99 Release 1.6.8 on 02/21/2014
101 * Add a tool to dump BDB data to Text and the reverse
102 * Fix racing condition problem in CachingStoreClientFactory
103 * Testing, client shell and admin tool enhancements from suletm,
104 ahimta, bhsaurabh and Holden Caufield
105 * ConsistencyCheckTool enhancement
106 * Fix bug in Coordinator for enabling different startup ports
108 Release 1.6.7 on 02/07/2014
110 * Updating the ivy file to refer to tusk-0.0.2 (build fix for nuage)
112 Release 1.6.6 on 02/07/2014
114 * Quota management in Voldemort
115 * Creation of read only stores in RO2 format
116 * Updation of query key feature in admin tool
118 Release 1.6.5 on 02/04/2014
120 * Adding AdminClient functionality to update store definitions
122 Release 1.6.4 on 02/03/2014
124 * Fix bug when RO server NPE when rolled back
125 * RO bug fix caused by RW optimization code
127 Release 1.6.3 on 01/24/2014
129 * Fix KeyVersionFetcher to not depend on nodeid 0
131 Release 1.6.2 on 01/10/2014
133 * Slops stored for nodes no longer in cluster, will die automatically
134 * Admin command to selectively purge slops destined to nodes/zones/stores
135 * Admin command to atomically update cluster.xml, stores.xml in one go
136 * Added admin command show-routing-plan to trace routing table for a key
137 * ZoneClipperCLI also adjusts store definitions based on dropped zone
138 * Numerous tests around non contiguous node ids/zone ids
140 Release 1.6.1 on 12/13/2013
142 * Avro support for fetch keys/entries admin command
143 * Store Clients are now cached by default, within the same factory
144 * Rigoruous tests around non contiguous node ids/zone ids
145 * ZenStoreClient efficient resource managerment
148 Release 1.6.0 on 11/26/2013
150 NOTE: This is an open source release! This release can be downloaded here:
151 https://github.com/voldemort/voldemort/releases
153 Changes made since 1.5.9:
155 * Fixes to remove dependency on non contiguous node ids
156 * Stats for store client
159 Release 1.5.9 on 11/05/2013
161 * Bug fix in AsyncMetadataVersionManager (related to store version tracking)
163 Release 1.5.8 on 10/29/2013
165 * Replace resthdfs jar with tusk.jar
166 * Incorporate retries into rest hdfs fetcher
168 Release 1.5.7 on 10/28/2013
170 * Voldemort now uses BDB 5.0.88
171 * Support for RO fetches via RestHdfs
172 * Fix NPE in Streaming Client
173 * Retrying fetches on all exceptions w/ delay
175 Release 1.5.6 on 10/18/2013
177 * Removed getStoreClientFactoryStats from StoreClientFactory interface (fix)
179 Release 1.5.5 on 10/18/2013
181 * Client shell with avro support
182 * Fix bind exceptions
183 * Fix ObsoleteVersion Exceptions
184 * Build once and push to multiple clusters
185 * Improvements to failure detector
187 * Prevent NPE in worker thread when shutdown
188 * Bug fixes in repair and prune job
190 Release 1.5.3 on 09/19/2013
192 * Bug fixes (NPE during shutdown
193 * Client side visibility changes (Histogram, Connect exception)
194 * Adding PruneJob and timebased resolving mechanism to streaming client
195 * R2Store cleanup required for the rest client
196 * Monitoring improvements (aggregated BDB stats, streaming stats)
197 * Monitoring and Config improvements for REST service
198 * Removed donor based rebalancing code
199 * Removed replica type from the code base
200 * Improvements to Rebalance controller
202 Release 1.4.7 on 09/04/13
205 * Added the Zone affinity feature on the client side
206 * Refactoring the rest package (Rest server and Coordinator)
207 * Using standard Netty pipeline in the Coordinator
208 * Added monitoring hooks for the Rest Service
210 Release 1.4.6 on 07/24/2013
213 * Cleanup of R2Store to work with the REST server
214 * Modified ZenStoreClient to avoid creating SystemStores during each
216 * Added slop streaming functionality
218 Release 1.4.4 on 07/08/2013
220 Changes made since 1.4.3
221 * Change Repartioner to swap among nodes from all specified zones
222 * Added rebalance shuffle, cluster expansion scripts
224 Release 1.4.3 on 07/03/2013
226 Note: Server changes are not backwards compatible. To use new
227 rebalancing tooling, servers must be upgraded before hand.
230 - Proxying / abortable rebalance
231 - added proxy puts which rebalance safely abortable
232 - improved proxy gets to check locally first and only fetch remote
234 - Repartitioning: improved algorithms to optimize partition layout
235 - Plan: avoids cross-zone data movement in most rebalance cases. Only
236 expanding into a new zone requires data movement across zones.
238 - better / more accurate progress monitoring.
239 - 'proxy pause' before rebalancing. Ensures clients pick up new
240 metadata and allows performance with proxy'ing to be observed
241 before servers start rebalancing.
242 - NOTE: Donor-based rebalance is currently broken by these changes.
244 - PartitionBalanceCLI: more nuanced and verbose analysis of cluster
246 - RepartitionerCLI: Stand alone tool to determine repartitioning
247 - Stand alone 'repartitioning' scripts for key use cases: new cluster,
248 shuffle (existing cluster), cluster expansion, and zone expansion.
249 - RebalancePlanCLI: Stand alone plan tool to determine cost of a
251 - RebalanceControllerCLI: Stand-alone tool for executing a specific
255 - moved to Guava from Google Collections
256 - better unit test coverage of zoned clusters
257 - added example prod configuration
258 - added tools directory
259 - .gitignore improvements
261 - fixed concurrency bugs in metadata store
262 - fixed vector clock comparisons
264 - write slops in more cases when they are needed
265 - KeySampler/Fetcher handle larger working sets with OOME
267 Changes made since 1.4.2
269 * Faster creation of BaseStoreRoutingPlan objects in the fast rebalancing path
271 Release 1.4.2 on 06/27/2013
272 * Fixing costly StoreRoutingPlan object construction
274 Release 1.4.0 on 06/21/2013
275 * Zone expansion release
277 Release 1.3.4 on 06/19/2013
278 * Read Write store bug fixes
279 - Rewrite of InMemoryStorageEngine + config to control multiVersionPuts
280 - Fixed a bug in the read-repair logic which was causing unnecessary puts.
281 - Fixed put operation in PipelineRoutedStore to ensure slop is submitted
282 - Fixed VectorClock bug triggered during versioned puts
283 * Rebalance improvement
284 - Made it safe to abort rebalance
285 - Proxy puts are established so that 'old' partition is updated and so an aborted rebalance can "roll back" without any data loss
286 - Proxy gets & puts are established within the zone making improving performance during rebalance
288 - Added monitoring capability in the Coordinator
290 Release 1.3.3 on 04/24/2013
291 * VoldemortBuildandPush
292 - Fixed bug with schema check
294 - Fixed issue with redundant callback invocation
295 Release 1.3.1 on 03/25/2013
297 - Fixed the bug in calculating checksums when we entere a retry loop
298 - refactored per file checksums
299 - added junit test case to simulate intermittent IO exceptions
300 * voldemort.client.protocol.admin.AdminClient
301 - Added AdminStoreClient so that AdminClient can do store operations
302 against specific store on a specific node.
303 - Added helper methods for diong put & get for specific node/store
304 - Added voldemort.client.protocol.admin.QueryKeyResult type to
305 simplify QueryKey interface
306 * Improved FetchStreamRequestHandler and sub-classes
307 - Renamed all sub-classes: 'FullScan' and 'PartitionScan' prefixes
308 for pertinent stream request handler implementations.
309 - Removed unused skipRecords parameter.
310 - Added recordsPerPartition parameter to fetch limited portion of
312 - All logic about how many keys to fetch (return to invoker) is
315 - Added many options to help improve the balance of (zoned) clusters.
316 - Analysis of the balance of a cluster is significantly more detailed.
317 - Fixed a bug that reduced the balance of a cluster each time it was
319 - Many different algorithms for improving cluster balance are
320 implemented in voldemort.utils.RebalanceClusterUtils
321 * ConsistencyCheck & ConsistencyFixCLI
322 - New tools for ensuring data durability. These tools are necessary
323 because slop creation can fail during put operations.
324 - ConsistencyCheck determines which keys, if any, lack
325 "consistency". I.e., are present on only a subset of the expected
327 - ConsistencyFix takes a list of bad (inconsistent) keys and makes
328 sure they are present on all expected partitions.
329 - ConsistencyFix also has an interface for repairing "orphaned" keys
330 that could result from an aborted rebalance.
331 * KeySamplerCLI & KeyVersionFetcherCLI
332 - KeySamplerCLI is a new tool that reads some number of keys for
333 specified partitions/stores.
334 - KeyVersionFetcherCLI is a new tool that, given a key and a store,
335 fetches the version from all nodes that host a partition that
336 ought to store a replica of the key's value.
337 - Together, KeySamplerCLI and KeyVersionFetcherCLI correctly
338 implement the intended functionality of the Entropy tool (for
339 servers that implement either FullScan and PartitionScan fetches).
340 - Entropy tool had been used in the past to verify a sample of keys
341 before and after a rebalance. Entropy tool does not work as
342 intended/expected. This is exacerbated by the partition aware
343 layouts. Instead of trying to fix the Entropy tool, these two new
344 tools were developed. Entropy is deprecated and will eventually be
345 removed from the code base.
346 * Substantial refactoring of helper & util methods:
347 - voldemort.cluster.Cluster : added helper methods
348 - voldemort.utils.ClusterInstance : wraps up one Cluster &
349 List<StoreDefinition>
350 - voldemort.utils.Cluster : utils for single Cluster object.
351 - voldemort.utils.NodeUtils : utils for Node object.
352 - voldemort.utils.RebalanceUtils : Many methods moved to more
353 appropriate helper classes
354 - voldemort.utils.StoreDefinitionUtils : utils for StoreDefinition
356 - voldemort.utils.StoreInstance : wraps up one Cluster & one
359 - ByteUtils toHexString & from HexString now rely on standard
361 - voldemort.client.AdminFetchTest now tests FullScan and
362 PartitionScan fetches
363 - voldemort.store.routed.ReadRepairerTest annotated all tests with
367 Release 1.3.0 on 03/08/2013
369 NOTE: This is an open source release! This release can be downloaded here:
370 http://github.com/voldemort/voldemort/downloads.
372 Changes made since 1.2.3
373 * VoldemortConfig and ClientConfig now contain detailed documentation
374 * BDB-JE defaults set to ones in prod@linkedin
375 * Bug fixes on kerberos support for Hadoop
378 Release 1.2.3 on 02/20/2013
380 Changes made since 1.2.2
381 * Added a retry loop and synchronized block while getting Hadoop FS
382 * Code cleanup in HdfsFetcher to make it more readable.
383 * Throwing explicit exceptions in HdfsFetcher instead of
384 returning null to be more precise in the Azkaban logs.
387 Release 1.2.2 on 02/19/2013
389 Changes made since 1.2.1
390 * Synchronized the streaming API
391 * Fixed some of the streaming API tests.
394 Release 1.2.1 on 0/30/2013
396 Changes made since 1.2.0
397 * Added a Streaming API and related tests.
398 * Refactoring of the admin client apis into functional inner classes
401 Release 1.2.0 on 01/21/2013
403 Changes made since 1.1.9
404 * Added an Admin API to fetch orphaned key / entries
405 * Improved some tests related to streaming API.
406 * Correcting commons-codec version in ivy file (1.4)
409 Release 1.1.9 on 01/15/2013
411 Changes made since 1.1.8
412 * Asynchronous socket checkout improvements
413 * Changed checkout behavior of KeyedResourcePool to only create new
414 connections when there are no resources available (rather than
415 creating new connections until the pool is full)
416 * Changed QueuedKeyedResourcePool.reset behavior to better match
417 KeyedResourcePool (i.e., to not cancel queued asynchronous
418 requests unnecessarily)
419 * Removed (unnecessary) synchronization primitives from keyed resource pool
420 * Reduce granularity of failure detector locking within ThresholdFailureDetector
421 * Minor features/improvements
422 * Less verbose logging in the face of expected exceptions and errors
423 * Refactored (Queued)KeyedResourcePoolTest
425 * Fixed possible race condition for resource creation in KeyedResourcePool
426 * More efficient (time & space) and simpler Histogram implementation
430 Release 1.1.8 on 01/14/2013
432 Changes made since release 1.1.7
433 * Enhanced Server Monitoring
435 -- Streaming operations to the server
436 -- BDB storage exception counts
437 * Ability to turn off BDB Checkpointing during batch modifications
438 * Added ability to delete old checksum files in Build and Push reducer
439 * Upgrade Hadoop jar to 1.0.4-p2
442 Release 1.1.7 on 01/03/2013
444 NOTE: This release is based off of release 1.1.4
446 Changes made since release 1.1.4
447 * Upgrading Hadoop jar to 1.0.2
448 * Added support for Kerberos authentication in HdfsFetcher
449 * Extra config parameters for Kerberos config and keytab file
452 NOTE: Release 1.1.5 and 1.1.6 are special client side releases
453 not based off of master. 1.1.5 was rolled back to to a weird bug.
454 1.1.6 is a special client side release including Auto-
455 bootstrapper and Versioned Avro support.
458 Release 1.1.4 on 11/29/2012
460 Changes made since release 1.1.3
461 * Added BDB parameters to control LRU behavior in cache & proactive cleaner migration
462 * Added a mlock fix for pinning the indexes of RO stores in memory
465 Release 1.1.3 on 11/28/2012
467 Changes made since release 1.1.2
468 * Fixed a bug in the build and push job, specifically the Mapper
469 that caused collisions
470 * Added retry mechanism with the HDFS fetcher for hftp
473 Release 1.1.2 on 10/31/2012
475 Changes made since release 1.1.1
476 * Reverted a change to voldemort.versioning.Versioned.getVersion() so
477 that a Version is returned as our clients expect.
480 Release 1.1.1 on 10/30/2012
482 Changes made since release 1.1.0
483 * Fixed connection leak in ClientRequestExecutorFactory
484 * Changed client to default to DefaultStoreClient
487 Release 1.1.0 on 10/19/2012
489 Changes made since release 1.0.0
491 IMPORTANT NOTE : This release has significant changes to the BDB storage layer.
492 Users are required to read the bin/PREUPGRADE_FOR_1_1_X_README file
493 thoroughly before attempting to upgrade to 1.1.0. The necessary data
494 conversion will be done through bin/voldemort-convert-bdb.sh
496 * Upgrading to JE 4.1.17
497 * New data format that handles conflicting updates in Voldemort more
499 * Move data off heap and only use it for Index
500 * When scanning, evict whatever you bring in right away.
501 * Partition based scan api to dramatically speed up rebalancing & restore
502 using Partition aware scans (you exactly scan whatever you want to fetch)
503 * Flexible knobs to control scheduling of DataCleanupJob
506 Release 1.0.0 on 10/17/2012
508 NOTE: The large version number jump from 0.96 to 1.0.0 is to
509 standardize on a version number of the sort MAJOR.MINOR.PATCH. This
510 change is part of our effort to treat internal and open source
511 releases in a much more similar manner. Along these lines, release
512 notes for internal releases (like this one) are committed on the
513 master branch. We hope this improves transparency as we work towards
514 the next open source release.
516 Changes made since release 0.96
518 * Auto bootstrapping: ZenStoreClient and System stores
519 * Added server side system stores for managing metadata
520 * ZenStoreClient interacts with system stores
521 * ZenStoreClient auto bootstraps whenever cluster.xml or stores.xml changes
522 * Added a new routing strategy to route to all with local preference
523 * Added a client-registry for publishing client info and config values
524 * Updated LazyClientStore to try to bootstrap during Init
525 * Modified Failure Detector to work on a shared cluster object reference
526 * Avro: schema evolution and read only support
527 * Added new Avro serializer type that supports schema evolution
528 * Added Avro support to read only stores
529 * Added LinkedIn build-and-push Azkaban jobs to build read only stores to contrib
530 * Added a schema backwards compatibility check to VoldemortAdminTool and on server startup to prevent mishaps due to bad schemas
531 * Non-blocking IO: Fixed design flaw that blocked in the face of slow servers
532 * Asynchronous operations no longer do a blocking checkout to get a SocketDestination
533 * Added additional stats collection for better visibility into request queues
535 * Enhanced VoldemortAdminTool to update store metadata version
536 * Enhanced VoldemortAdminTool to work with the new system stores
537 * Added feature to voldemort-shell.sh to dump byte & object arrays
538 * Added a SlowStorageEngine for testing degraded mode performance
539 * Added mechanism to isolate BDB cache usage among stores
540 * Enhanced debug logging (for traffic analysis).
541 * Python client bug fixes (from pull request)
542 * Improved messages in request tracing
543 * Cleaned up help/usage messages within the client shell
544 * Added server config to control socket backlog
545 * Added "--query-keys" option to query multiple keys of multiple stores from specific node
546 * Added control to DataCleanupJob Frequency
547 * Unified jmxid as the factory across the board
549 * bin/generate_cluster_xml.py to generate cluster.xml
550 * bin/repeat-junit.sh and bin/repeat-junit-test.sh to repeatedly run tests
552 * Changed getall return behavior to comply with javadoc
553 * Fixed a bug that caused unnecessary serial requests in getall
554 * HFTP performance issue bug fix (fix in byte buffer and copy process)
555 * Fixed a bug that prevented "--fetch-keys" and "--fetch-entries" in admin tool from showing multiple store results
556 * Fixed problem in sample config that prevented the server from starting
557 * Fixed some intermittent BindException failures across many unit tests
558 * Fixed some intermittent rebalance test failures
559 * Wrapped long running tests with timeouts
562 Release 0.96 on 09/05/2012
564 Changes made since 0.90.1
567 * Append cluster name to various mbeans for better stats display
568 * Implement average throughput in bytes
570 * Add 95th and 99th latency tracking
571 * Add stats for ClientRequestExecutorPool
572 * Add error/exception count and max getall count
573 * BDB+ Data cleanup Monitoring changes
575 * Donor-based rebalancing and post cleanup (see https://github.com/voldemort/voldemort/wiki/Voldemort-Donor-Based-Rebalancing for more details)
576 * Rebalancing integration testing framework (under test/integration/voldemort/rebalance/)
577 * Generate multiple cluster.xml files based on the number specified when running the tool and choose the cluster with the smallest std dev as the final-cluster.xml
578 * Add status output to log for updateEntries (used by rebalancing)
579 * Read-only pipeline:
580 * Add hftp and webhdfs support
581 * Read-only bandwidth dynamic throttler
582 * Add minimum throttle limit per store
583 * Add rollback capability to the Admin tool
584 * Voldemort-backed stack and index linked list impl
585 * Change client requests to not process responses after timeout
586 * Modified client request executor timeout to not factor in the NIO selector timeout
587 * Added BDB native backup capabalities, checksum verification and incremental backups (well tested, but not yet used in production)
588 * Add additional client-side tracing for debugging and consistency analytics
589 * Clean up logging during exception at client-side
590 * Security exception handling
591 * Add snappy to CompressionStrategyFactory
592 * Add configurable option to interrupt service being unscheduled
593 * Add logging support for tracking ScanPermit owners (for debugging purposes)
594 * Add a jmx terminate operation for async jobs
595 * Add zone option for restore from replicas
596 * Changing the enable.nio.connector to true by default
597 * Better disconnection handling for python client
598 * Split junit tests into a long and a short test suites
599 * Add separate timeouts for different operations (put, get, delete, and getAll
600 * Allow getAll to return partial results upon timeout
601 * Improved cluster generation tool
602 * Added log4j properties folder for junit test
604 * httpclient 3.x to httpclient 4.x
605 * Fix NPE in listing read-only store versions
606 * Fixed 2 failure detector bugs during rebalancing or node swapping
607 * Fixed a thread leak issue in StreamingSlopPusher
609 * Fixed a bug in TimeBasedInconsistency resolver.
610 * Fixed race condition in client socket close
611 * Fixed a potential deadlock issue in ScanPermitWrapper
612 * Fixed a bug where a read returns null (on rare occations) when being concurrent with a write
613 * Fixed a performance bug in HdfsFetcher when hftp is used
616 Changes made since 0.90
618 * Updated the documentation for Voldemort shell tool in NOTES
619 * Added Admin API to perform Bdb data cleanup (repairJob)
620 and corresponding unit tests
621 * Fixes in restore from replication, store creation code
622 * Improved failure detector configuration. ThresholdFailureDetector
623 is now the default option
624 * Multiple Fixes to the Voldemort ruby client
625 * Added additional Jmx metrics to expose Bdb environment statistics,
626 caching statistics and Voldemort batch operation statistics
627 * Updated default timeout for restore 'from replica' to 365 days
628 * New feature in the performance tool: '--use-sample' option enables
629 'read, write back unmodified' transactions in place of writes in
630 the workload (allows for testing read-write transactions on stores
631 with complex schemas)
632 * Added the ability to dynamically update cluster.xml and
633 reinitialize the scheduler
635 Release 0.90 on 7/10/2011
637 Changes made since 0.81
638 * All upgrading instructions can be found here - https://github.com/voldemort/voldemort/wiki/Upgrading-from-0.81
640 - Single consolidated administrative tool - Got rid of the admin client
641 shell and have a better Voldemort admin tool. More documentation -
642 https://github.com/voldemort/voldemort/wiki/Voldemort-Admin-tool
645 - Basic GUI in JRuby ( and Sinatra ), capable of querying for store metadata.
646 - Read contrib/web-manager/README.md for more details
649 - New better rebalancing support with (a) checkpointing (b) progress bar (c) support for RO store rebalancing
650 - Documentation - https://github.com/voldemort/voldemort/wiki/Voldemort-Rebalancing
651 - In the process solves Issue 203, 288, 305, 307, 311
653 * Client side changes
654 - Pipeline routed store ( with support for topology awareness ) - Changed the default client side routing
655 to use a state machine like system. More documentation - https://github.com/voldemort/voldemort/wiki/RoutedStore-redesign
656 - Lazy store client - Some of our clients are very inactive and don't really do many requests. For such
657 clients it doesn't make sense to bootstrap the metadata at startup. Hence from now onwards clients instantiated
658 from SocketStoreClientFactory give a LazyStoreClient which will not bootstrap till the first request is made
659 - Caching store client factory - We have also checked-in a new store client factory which we use internally
660 at LinkedIn to cache store clients for stores which we repeatedly.
661 - Failure detector - We have fixed some bugs in the ThresholdFailureDetector. This is an improvement over the naive
662 BannageFailureDetector which would aggressively mark nodes down after a single failure. More details -
663 https://github.com/voldemort/voldemort/wiki/Client-side-failure-detector-implementations
664 - Issue 219: StoreClient::put returns a Version - You may have to upgrade your clients since now the StoreClient returns
665 a version to be used in successive requests.
667 * Read-only store changes
668 - Admin based store swapper - Till 0.81 we relied on a servlet based fetcher / swapper.
669 Now we support an admin based store swapper which reports progress.
670 - Other changes - (a) New directory format (b) Two new data format. Support for iterating over read-only data
671 (c) Checksum of data in .metadata file (d) Some monitoring changes of RO stores
674 - Introduction of JNA 3.2.7 - We have introduced JNA 3.2.7 in this release for supporting symbolic links in read-only stores.
675 More details about the new RO format and changes can be found here - https://github.com/voldemort/voldemort/wiki/Upgrading-from-0.81
676 - Protocol Buffers 2.3.0 - Upgrading protocol buffers from 2.2.0 to 2.3.0
677 - Avro 1.4.0 - Upgraded Avro from 1.3.0 to 1.4.0
680 - Added support for Krati 0.3.6 as a storage engine ( http://sna-projects.com/krati/ ) -
681 https://github.com/voldemort/voldemort/tree/release-090/contrib/krati
684 - Hinted handoff - https://github.com/voldemort/voldemort/wiki/Hinted-Handoff
685 - Experimental support for server side transforms - https://github.com/voldemort/voldemort/wiki/Server-side-transforms-in-Voldemort
686 - Topology awareness ( i.e. datacenter / rack ) - https://github.com/voldemort/voldemort/wiki/Topology-awareness-capability
687 - Repair Job ( Job which will delete data if the node is not responsible for it )
690 - Tons of JMX changes ( average bytes, etc ) - More details ( https://github.com/voldemort/voldemort/wiki/JMX-Monitoring )
691 - Key distribution generator - Ability to estimate skew of your cluster ( ./bin/run-class.sh voldemort.utils.KeyDistributionGenerator )
694 - Python - Updated Python client with support for Binary JSON serialized stores ( ./clients/python/README )
695 - Ruby - Updated Ruby client ( ./clients/ruby/README.md )
697 Release 0.81 on 6/15/2010
699 Changes made since 0.80.2:
701 * IMPORTANT: we have upgraded the Hadoop Core jar to v0.20.2. Since
702 this version of Hadoop requires Java 6, in order to retain backwards
703 compatibility with Java 5, you will need to replace this jar with
704 the Hadoop 0.18.* core jar.
705 * Multiple donors for Voldemort Rebalancing: speeds up rebalancing, by
706 allowing a single stealer node to transfer partitions from multiple
708 * Features for EC2 Testing
709 * Read-only Stores: backwards compatibility with 0.70, bug fix in
711 * Hadoop InputFormat, Pig LoadFunc in Contrib: ability to perform
712 Map/Reduce (either directly via Hadoop or using Pig) over data in
713 Voldemort stores. See:
714 < contrib/hadoop/test/voldemort/hadoop/VoldemortWordCount.java > for
716 * Voldemort Performance Tool: performance measurement tool based on
717 YCSB (Yahoo Cloud Serving Benchmark) code. Run
718 < bin/voldemort-performance-tool.sh --help > for more information.
719 * Reverted default failure detector implementation back to
720 BannagedPeriodFailureDetector due to potential bugs in the
721 ThresholdFailureDetector
723 Release 0.80.2 on 4/27/2010
725 Changes made since 0.80.1:
727 * Batched NIO writes (improves performance for AdminClient/Streaming
728 operations when the NIO connector is enabled)
729 * VoldemortAdminTool: Added support to specify stores, support to
730 fetch all keys to a binary file, support to save keys in ASCII
731 (JSON) format [experimental], capability to add stores. Run
732 < bin/voldemort-admin-tool.sh --help > for more information.
733 * Minor bug fixes for Rebalancing
734 * Issue 240: Voldemort fetcher should use different temp directories for
736 * Checksum capability during construction of Voldemort read-only stores
738 Release 0.80.1 on 3/23/2010
740 Changes made since 0.80:
742 * Issue 133: Support for Apache Avro as a serialization format
743 * Issue 223: Changed the default client to use
744 TresholdFailureDetector
745 * Fixed issue 222: Revised KeyedResourcePool.close(K key) to fix
747 * Fixed issue 198: Revised ReadRepairer to use a separate copy of the
748 vector clock, fixing a situation where NoSuchElementException would
750 * Miscellaneous enhancement: support for TCP keep-alives, improved
751 read only store utilities, command line interface to AdminClient,
752 improved load testing tools
754 Release 0.80 on 2/18/2010
756 Changes made since 0.70.1:
758 * IMPORTANT: backwards compatibility between the client and server has
759 changed. A backwards incompatibility in the wire protocol was found
760 between releases 0.60 to 0.70.1 and releases prior to 0.60. We chose
761 to make 0.80 compatible with 0.57.1 and earlier versions, while
762 introducing an incompatibility with 0.60-0.70.1. What this means is
763 that if you're presently running 0.60 and higher, you would need to
764 upgrade the Voldemort jar files on *all* servers and clients.
765 * Upgraded the BDB storage engine to use BerkeleyDB-JE 4.0.92,
766 retaining ability to use BerkeleyDB-JE 3.3.* if desired. IMPORTANT:
767 if one switches from BerkeleyDB-JE 3.3.* to 4.0.92 they will be able
768 to access all of existing data. Once a switch has been made to
769 4.0.92 the data will not be readable by earlier versions of BDB. If
770 there's a chance that a roll back to 3.3.* might be needed, the best
771 course of action will be to make a backup of existing data
773 Switching between 3.3.* and 4.0.* would also require rebuilding the
774 class files (e.g., by running "ant clean && ant release" after
775 replacing the BDB jar files).
776 * Compression support for read-only stores
777 * Increased the socket buffer size for transferring read-only
778 stores from Hadoop for improved performance over high-latency links
779 * NIO support for the Admin Service, including Streaming
781 * Support for adding stores on the fly via the Admin
783 * Fixed issue 209: Incorrect object passed to List.contains in
784 RebalanceUtils.getLatestCluster()
785 * Fixed issue 211: Unnecessary read repairs during getAll() with more
787 * Other enhancements: better CLI for rebalancing, throttling in
788 Admin Service is now based on all disk activity
790 Release 0.70.1 on 2/1/2010
792 Changes made since 0.70:
794 * Fixed issue 205: if no keys passed to getAll() were in partitions
795 undergoing rebalancing, proxyGetAll() would be called with an
796 empty list even if rebalancing wasn't happening
798 Release 0.70 on 1/27/2010
800 Changes made since 0.60.1:
802 * A beta of rebalancing (dynamic cluster expansion) support merged
803 into the main branch. See the project's wiki for more information:
804 http://wiki.github.com/voldemort/voldemort/voldemort-rebalancing
805 * New failure detector merged into the main branch:
806 http://wiki.github.com/voldemort/voldemort/failure-detection
807 * Beta mechanism for restoring all of node's data from replicas on
808 demand. This is an alternative to a more gradual mechanism provided
809 by read-repair: useful when a machine is down for a prolonged period
810 and is then re-inserted into the cluster.
811 Invoked via JMX: the operation is restoreDataFromReplication in the
812 voldemort.server.VoldemortServer MBean, with a mandatory parameter
813 (integer >= 1) indicating the number of transfers to do in parallel.
814 * Simple gossip protocol (for cluster metadata) merged
815 into the main branch. Disabled by default: use "enable.gossip=true"
816 to enable, use "gossip.interval.ms" to set an interval at which gossip
817 occurs (default: 30000 i.e., 30 seconds).
818 * Fixed issue 190: add a way of aggregating performance data over
820 * Fixed issue 181: stack trackes shouldn't be filled for Obsolete
823 Release 0.60.1 on 12/18/2009
825 Minor changes made since 0.60:
827 * Better logging in the exception thrown if config/.temp and
828 config/.version are copied
829 * Bumping up the version to 0.60.1 in order to release updated
830 archives, fixing an error in the stores.xml for single_node_cluster
833 Release 0.60 on 12/15/2009
835 Changes made since 0.57.1:
837 * Admin Client/Server API: adds support for streaming-based transfer
838 of entries between nodes, deleting entries on remote nodes,
839 remotely deleting and updating metadata
840 * EC2 testing: a way to periodically run integration and performance
841 tests which involve Voldemort instances on different machines
842 * Experimental support for views
843 * Interpolation search for read-only stores
844 * Support for large lists and strings in the JSON serializer
845 * LZF compression support
846 * Ruby client contributed
847 * Fixed issue 170: hanging if a port is used by another process
848 * Fixed issue 122: suspicious integer division in
849 RequestCounter.getThroughput
850 * Miscellaneous improvements and bug fixes for read-only stores
851 * Fixed issue 168: added StorageEngine.keys()
853 Release 0.57.1 on 11/27/2009
855 Minor change made since 0.57:
857 * Modified build.xml to exclude .git directory from release tarballs/zipfiles
859 Release 0.57 on 11/16/2009
861 The following changes were made since 0.56:
863 * Fixed an issue in ReadOnlyEngine's close() method
864 * Fixed hidden logging in StorageService
865 * Fix for issue 163 (lock mode during get)
866 * Exposed bdb environment stats with setFast(false)
868 Release 0.56 on 10/26/2009
870 The following changes were made since 0.55:
872 * Fix for issue 164: Changed default bdb.max.logfile.size to 60MB
873 * Make file deletes asynchronous in read only store swap
874 * Added better debug logs for bdb stats
875 * Fixed race condition in AbstractSocketPoolTest
876 * Added improved monitoring for bdb stats
877 * Added backoff and retry logic in bootstrap code
878 * Not logging obosleteVersionException(s) or counting it in JMX exception count
880 * C++ client building on OS X
881 * Rely only on the number of versions returned to decide whether to retrieve
882 the value for put(K,V)
883 * Implemented issue 152: getVersion() API in Store
885 Release 0.55 on 10/7/2009
887 The following changes were made since 0.52 (in summary):
889 * Add an event throttler
890 * Added DataCleanupJob
891 * Protocol buffers at version 2.2.0
892 * BDB JE upgraded to 3.3.87
893 * Added data compression support (enable by adding
897 to value-serializer or key-serializer sections in stores.xml)
898 * Added a resource pool/socket pool implementation (no longer using
900 * Added server-side NIO support (enabled by setting
901 enable.nio.connector=true
902 in server.properties)
903 * Improved the efficiency of the protocol buffers network protocol by using
904 CodedOutputStream/CodedInputStream (zero copy transfers)
905 * Fix for issue #21: incorrectness in vector clock inconsistency resolver
906 * Upgrade google-collections
907 * Support for building read only stores in Hadoop