1 Release 1.3.0 on 03/08/2013
3 NOTE: This is an open source release! This release can be downloaded here:
4 http://github.com/voldemort/voldemort/downloads.
6 Changes made since 1.2.3
7 * VoldemortConfig and ClientConfig now contain detailed documentation
8 * BDB-JE defaults set to ones in prod@linkedin
9 * Bug fixes on kerberos support for Hadoop
11 Release 1.2.3 on 02/20/2013
13 Changes made since 1.2.2
14 * Added a retry loop and synchronized block while getting Hadoop FS
15 * Code cleanup in HdfsFetcher to make it more readable.
16 * Throwing explicit exceptions in HdfsFetcher instead of
17 returning null to be more precise in the Azkaban logs.
19 Release 1.2.2 on 02/19/2013
21 Changes made since 1.2.1
22 * Synchronized the streaming API
23 * Fixed some of the streaming API tests.
25 Release 1.2.1 on 0/30/2013
27 Changes made since 1.2.0
28 * Added a Streaming API and related tests.
29 * Refactoring of the admin client apis into functional inner classes
31 Release 1.2.0 on 01/21/2013
33 Changes made since 1.1.9
34 * Added an Admin API to fetch orphaned key / entries
35 * Improved some tests related to streaming API.
36 * Correcting commons-codec version in ivy file (1.4)
38 Release 1.1.9 on 01/15/2013
40 Changes made since 1.1.8
41 * Asynchronous socket checkout improvements
42 * Changed checkout behavior of KeyedResourcePool to only create new
43 connections when there are no resources available (rather than
44 creating new connections until the pool is full)
45 * Changed QueuedKeyedResourcePool.reset behavior to better match
46 KeyedResourcePool (i.e., to not cancel queued asynchronous
47 requests unnecessarily)
48 * Removed (unnecessary) synchronization primitives from keyed resource pool
49 * Reduce granularity of failure detector locking within ThresholdFailureDetector
50 * Minor features/improvements
51 * Less verbose logging in the face of expected exceptions and errors
52 * Refactored (Queued)KeyedResourcePoolTest
54 * Fixed possible race condition for resource creation in KeyedResourcePool
55 * More efficient (time & space) and simpler Histogram implementation
59 Release 1.1.8 on 01/14/2013
61 Changes made since release 1.1.7
62 * Enhanced Server Monitoring
64 -- Streaming operations to the server
65 -- BDB storage exception counts
66 * Ability to turn off BDB Checkpointing during batch modifications
67 * Added ability to delete old checksum files in Build and Push reducer
68 * Upgrade Hadoop jar to 1.0.4-p2
71 Release 1.1.7 on 01/03/2013
73 NOTE: This release is based off of release 1.1.4
75 Changes made since release 1.1.4
76 * Upgrading Hadoop jar to 1.0.2
77 * Added support for Kerberos authentication in HdfsFetcher
78 * Extra config parameters for Kerberos config and keytab file
81 NOTE: Release 1.1.5 and 1.1.6 are special client side releases
82 not based off of master. 1.1.5 was rolled back to to a weird bug.
83 1.1.6 is a special client side release including Auto-
84 bootstrapper and Versioned Avro support.
87 Release 1.1.4 on 11/29/2012
89 Changes made since release 1.1.3
90 * Added BDB parameters to control LRU behavior in cache & proactive cleaner migration
91 * Added a mlock fix for pinning the indexes of RO stores in memory
94 Release 1.1.3 on 11/28/2012
96 Changes made since release 1.1.2
97 * Fixed a bug in the build and push job, specifically the Mapper
98 that caused collisions
99 * Added retry mechanism with the HDFS fetcher for hftp
102 Release 1.1.2 on 10/31/2012
104 Changes made since release 1.1.1
105 * Reverted a change to voldemort.versioning.Versioned.getVersion() so
106 that a Version is returned as our clients expect.
109 Release 1.1.1 on 10/30/2012
111 Changes made since release 1.1.0
112 * Fixed connection leak in ClientRequestExecutorFactory
113 * Changed client to default to DefaultStoreClient
116 Release 1.1.0 on 10/19/2012
118 Changes made since release 1.0.0
120 IMPORTANT NOTE : This release has significant changes to the BDB storage layer.
121 Users are required to read the bin/PREUPGRADE_FOR_1_1_X_README file
122 thoroughly before attempting to upgrade to 1.1.0. The necessary data
123 conversion will be done through bin/voldemort-convert-bdb.sh
125 * Upgrading to JE 4.1.17
126 * New data format that handles conflicting updates in Voldemort more
128 * Move data off heap and only use it for Index
129 * When scanning, evict whatever you bring in right away.
130 * Partition based scan api to dramatically speed up rebalancing & restore
131 using Partition aware scans (you exactly scan whatever you want to fetch)
132 * Flexible knobs to control scheduling of DataCleanupJob
135 Release 1.0.0 on 10/17/2012
137 NOTE: The large version number jump from 0.96 to 1.0.0 is to
138 standardize on a version number of the sort MAJOR.MINOR.PATCH. This
139 change is part of our effort to treat internal and open source
140 releases in a much more similar manner. Along these lines, release
141 notes for internal releases (like this one) are committed on the
142 master branch. We hope this improves transparency as we work towards
143 the next open source release.
145 Changes made since release 0.96
147 * Auto bootstrapping: ZenStoreClient and System stores
148 * Added server side system stores for managing metadata
149 * ZenStoreClient interacts with system stores
150 * ZenStoreClient auto bootstraps whenever cluster.xml or stores.xml changes
151 * Added a new routing strategy to route to all with local preference
152 * Added a client-registry for publishing client info and config values
153 * Updated LazyClientStore to try to bootstrap during Init
154 * Modified Failure Detector to work on a shared cluster object reference
155 * Avro: schema evolution and read only support
156 * Added new Avro serializer type that supports schema evolution
157 * Added Avro support to read only stores
158 * Added LinkedIn build-and-push Azkaban jobs to build read only stores to contrib
159 * Added a schema backwards compatibility check to VoldemortAdminTool and on server startup to prevent mishaps due to bad schemas
160 * Non-blocking IO: Fixed design flaw that blocked in the face of slow servers
161 * Asynchronous operations no longer do a blocking checkout to get a SocketDestination
162 * Added additional stats collection for better visibility into request queues
164 * Enhanced VoldemortAdminTool to update store metadata version
165 * Enhanced VoldemortAdminTool to work with the new system stores
166 * Added feature to voldemort-shell.sh to dump byte & object arrays
167 * Added a SlowStorageEngine for testing degraded mode performance
168 * Added mechanism to isolate BDB cache usage among stores
169 * Enhanced debug logging (for traffic analysis).
170 * Python client bug fixes (from pull request)
171 * Improved messages in request tracing
172 * Cleaned up help/usage messages within the client shell
173 * Added server config to control socket backlog
174 * Added "--query-keys" option to query multiple keys of multiple stores from specific node
175 * Added control to DataCleanupJob Frequency
176 * Unified jmxid as the factory across the board
178 * bin/generate_cluster_xml.py to generate cluster.xml
179 * bin/repeat-junit.sh and bin/repeat-junit-test.sh to repeatedly run tests
181 * Changed getall return behavior to comply with javadoc
182 * Fixed a bug that caused unnecessary serial requests in getall
183 * HFTP performance issue bug fix (fix in byte buffer and copy process)
184 * Fixed a bug that prevented "--fetch-keys" and "--fetch-entries" in admin tool from showing multiple store results
185 * Fixed problem in sample config that prevented the server from starting
186 * Fixed some intermittent BindException failures across many unit tests
187 * Fixed some intermittent rebalance test failures
188 * Wrapped long running tests with timeouts
191 Release 0.96 on 09/05/2012
193 Changes made since 0.90.1
196 * Append cluster name to various mbeans for better stats display
197 * Implement average throughput in bytes
199 * Add 95th and 99th latency tracking
200 * Add stats for ClientRequestExecutorPool
201 * Add error/exception count and max getall count
202 * BDB+ Data cleanup Monitoring changes
204 * Donor-based rebalancing and post cleanup (see https://github.com/voldemort/voldemort/wiki/Voldemort-Donor-Based-Rebalancing for more details)
205 * Rebalancing integration testing framework (under test/integration/voldemort/rebalance/)
206 * Generate multiple cluster.xml files based on the number specified when running the tool and choose the cluster with the smallest std dev as the final-cluster.xml
207 * Add status output to log for updateEntries (used by rebalancing)
208 * Read-only pipeline:
209 * Add hftp and webhdfs support
210 * Read-only bandwidth dynamic throttler
211 * Add minimum throttle limit per store
212 * Add rollback capability to the Admin tool
213 * Voldemort-backed stack and index linked list impl
214 * Change client requests to not process responses after timeout
215 * Modified client request executor timeout to not factor in the NIO selector timeout
216 * Added BDB native backup capabalities, checksum verification and incremental backups (well tested, but not yet used in production)
217 * Add additional client-side tracing for debugging and consistency analytics
218 * Clean up logging during exception at client-side
219 * Security exception handling
220 * Add snappy to CompressionStrategyFactory
221 * Add configurable option to interrupt service being unscheduled
222 * Add logging support for tracking ScanPermit owners (for debugging purposes)
223 * Add a jmx terminate operation for async jobs
224 * Add zone option for restore from replicas
225 * Changing the enable.nio.connector to true by default
226 * Better disconnection handling for python client
227 * Split junit tests into a long and a short test suites
228 * Add separate timeouts for different operations (put, get, delete, and getAll
229 * Allow getAll to return partial results upon timeout
230 * Improved cluster generation tool
231 * Added log4j properties folder for junit test
233 * httpclient 3.x to httpclient 4.x
234 * Fix NPE in listing read-only store versions
235 * Fixed 2 failure detector bugs during rebalancing or node swapping
236 * Fixed a thread leak issue in StreamingSlopPusher
238 * Fixed a bug in TimeBasedInconsistency resolver.
239 * Fixed race condition in client socket close
240 * Fixed a potential deadlock issue in ScanPermitWrapper
241 * Fixed a bug where a read returns null (on rare occations) when being concurrent with a write
242 * Fixed a performance bug in HdfsFetcher when hftp is used
245 Changes made since 0.90
247 * Updated the documentation for Voldemort shell tool in NOTES
248 * Added Admin API to perform Bdb data cleanup (repairJob)
249 and corresponding unit tests
250 * Fixes in restore from replication, store creation code
251 * Improved failure detector configuration. ThresholdFailureDetector
252 is now the default option
253 * Multiple Fixes to the Voldemort ruby client
254 * Added additional Jmx metrics to expose Bdb environment statistics,
255 caching statistics and Voldemort batch operation statistics
256 * Updated default timeout for restore 'from replica' to 365 days
257 * New feature in the performance tool: '--use-sample' option enables
258 'read, write back unmodified' transactions in place of writes in
259 the workload (allows for testing read-write transactions on stores
260 with complex schemas)
261 * Added the ability to dynamically update cluster.xml and
262 reinitialize the scheduler
264 Release 0.90 on 7/10/2011
266 Changes made since 0.81
267 * All upgrading instructions can be found here - https://github.com/voldemort/voldemort/wiki/Upgrading-from-0.81
269 - Single consolidated administrative tool - Got rid of the admin client
270 shell and have a better Voldemort admin tool. More documentation -
271 https://github.com/voldemort/voldemort/wiki/Voldemort-Admin-tool
274 - Basic GUI in JRuby ( and Sinatra ), capable of querying for store metadata.
275 - Read contrib/web-manager/README.md for more details
278 - New better rebalancing support with (a) checkpointing (b) progress bar (c) support for RO store rebalancing
279 - Documentation - https://github.com/voldemort/voldemort/wiki/Voldemort-Rebalancing
280 - In the process solves Issue 203, 288, 305, 307, 311
282 * Client side changes
283 - Pipeline routed store ( with support for topology awareness ) - Changed the default client side routing
284 to use a state machine like system. More documentation - https://github.com/voldemort/voldemort/wiki/RoutedStore-redesign
285 - Lazy store client - Some of our clients are very inactive and don't really do many requests. For such
286 clients it doesn't make sense to bootstrap the metadata at startup. Hence from now onwards clients instantiated
287 from SocketStoreClientFactory give a LazyStoreClient which will not bootstrap till the first request is made
288 - Caching store client factory - We have also checked-in a new store client factory which we use internally
289 at LinkedIn to cache store clients for stores which we repeatedly.
290 - Failure detector - We have fixed some bugs in the ThresholdFailureDetector. This is an improvement over the naive
291 BannageFailureDetector which would aggressively mark nodes down after a single failure. More details -
292 https://github.com/voldemort/voldemort/wiki/Client-side-failure-detector-implementations
293 - Issue 219: StoreClient::put returns a Version - You may have to upgrade your clients since now the StoreClient returns
294 a version to be used in successive requests.
296 * Read-only store changes
297 - Admin based store swapper - Till 0.81 we relied on a servlet based fetcher / swapper.
298 Now we support an admin based store swapper which reports progress.
299 - Other changes - (a) New directory format (b) Two new data format. Support for iterating over read-only data
300 (c) Checksum of data in .metadata file (d) Some monitoring changes of RO stores
303 - Introduction of JNA 3.2.7 - We have introduced JNA 3.2.7 in this release for supporting symbolic links in read-only stores.
304 More details about the new RO format and changes can be found here - https://github.com/voldemort/voldemort/wiki/Upgrading-from-0.81
305 - Protocol Buffers 2.3.0 - Upgrading protocol buffers from 2.2.0 to 2.3.0
306 - Avro 1.4.0 - Upgraded Avro from 1.3.0 to 1.4.0
309 - Added support for Krati 0.3.6 as a storage engine ( http://sna-projects.com/krati/ ) -
310 https://github.com/voldemort/voldemort/tree/release-090/contrib/krati
313 - Hinted handoff - https://github.com/voldemort/voldemort/wiki/Hinted-Handoff
314 - Experimental support for server side transforms - https://github.com/voldemort/voldemort/wiki/Server-side-transforms-in-Voldemort
315 - Topology awareness ( i.e. datacenter / rack ) - https://github.com/voldemort/voldemort/wiki/Topology-awareness-capability
316 - Repair Job ( Job which will delete data if the node is not responsible for it )
319 - Tons of JMX changes ( average bytes, etc ) - More details ( https://github.com/voldemort/voldemort/wiki/JMX-Monitoring )
320 - Key distribution generator - Ability to estimate skew of your cluster ( ./bin/run-class.sh voldemort.utils.KeyDistributionGenerator )
323 - Python - Updated Python client with support for Binary JSON serialized stores ( ./clients/python/README )
324 - Ruby - Updated Ruby client ( ./clients/ruby/README.md )
326 Release 0.81 on 6/15/2010
328 Changes made since 0.80.2:
330 * IMPORTANT: we have upgraded the Hadoop Core jar to v0.20.2. Since
331 this version of Hadoop requires Java 6, in order to retain backwards
332 compatibility with Java 5, you will need to replace this jar with
333 the Hadoop 0.18.* core jar.
334 * Multiple donors for Voldemort Rebalancing: speeds up rebalancing, by
335 allowing a single stealer node to transfer partitions from multiple
337 * Features for EC2 Testing
338 * Read-only Stores: backwards compatibility with 0.70, bug fix in
340 * Hadoop InputFormat, Pig LoadFunc in Contrib: ability to perform
341 Map/Reduce (either directly via Hadoop or using Pig) over data in
342 Voldemort stores. See:
343 < contrib/hadoop/test/voldemort/hadoop/VoldemortWordCount.java > for
345 * Voldemort Performance Tool: performance measurement tool based on
346 YCSB (Yahoo Cloud Serving Benchmark) code. Run
347 < bin/voldemort-performance-tool.sh --help > for more information.
348 * Reverted default failure detector implementation back to
349 BannagedPeriodFailureDetector due to potential bugs in the
350 ThresholdFailureDetector
352 Release 0.80.2 on 4/27/2010
354 Changes made since 0.80.1:
356 * Batched NIO writes (improves performance for AdminClient/Streaming
357 operations when the NIO connector is enabled)
358 * VoldemortAdminTool: Added support to specify stores, support to
359 fetch all keys to a binary file, support to save keys in ASCII
360 (JSON) format [experimental], capability to add stores. Run
361 < bin/voldemort-admin-tool.sh --help > for more information.
362 * Minor bug fixes for Rebalancing
363 * Issue 240: Voldemort fetcher should use different temp directories for
365 * Checksum capability during construction of Voldemort read-only stores
367 Release 0.80.1 on 3/23/2010
369 Changes made since 0.80:
371 * Issue 133: Support for Apache Avro as a serialization format
372 * Issue 223: Changed the default client to use
373 TresholdFailureDetector
374 * Fixed issue 222: Revised KeyedResourcePool.close(K key) to fix
376 * Fixed issue 198: Revised ReadRepairer to use a separate copy of the
377 vector clock, fixing a situation where NoSuchElementException would
379 * Miscellaneous enhancement: support for TCP keep-alives, improved
380 read only store utilities, command line interface to AdminClient,
381 improved load testing tools
383 Release 0.80 on 2/18/2010
385 Changes made since 0.70.1:
387 * IMPORTANT: backwards compatibility between the client and server has
388 changed. A backwards incompatibility in the wire protocol was found
389 between releases 0.60 to 0.70.1 and releases prior to 0.60. We chose
390 to make 0.80 compatible with 0.57.1 and earlier versions, while
391 introducing an incompatibility with 0.60-0.70.1. What this means is
392 that if you're presently running 0.60 and higher, you would need to
393 upgrade the Voldemort jar files on *all* servers and clients.
394 * Upgraded the BDB storage engine to use BerkeleyDB-JE 4.0.92,
395 retaining ability to use BerkeleyDB-JE 3.3.* if desired. IMPORTANT:
396 if one switches from BerkeleyDB-JE 3.3.* to 4.0.92 they will be able
397 to access all of existing data. Once a switch has been made to
398 4.0.92 the data will not be readable by earlier versions of BDB. If
399 there's a chance that a roll back to 3.3.* might be needed, the best
400 course of action will be to make a backup of existing data
402 Switching between 3.3.* and 4.0.* would also require rebuilding the
403 class files (e.g., by running "ant clean && ant release" after
404 replacing the BDB jar files).
405 * Compression support for read-only stores
406 * Increased the socket buffer size for transferring read-only
407 stores from Hadoop for improved performance over high-latency links
408 * NIO support for the Admin Service, including Streaming
410 * Support for adding stores on the fly via the Admin
412 * Fixed issue 209: Incorrect object passed to List.contains in
413 RebalanceUtils.getLatestCluster()
414 * Fixed issue 211: Unnecessary read repairs during getAll() with more
416 * Other enhancements: better CLI for rebalancing, throttling in
417 Admin Service is now based on all disk activity
419 Release 0.70.1 on 2/1/2010
421 Changes made since 0.70:
423 * Fixed issue 205: if no keys passed to getAll() were in partitions
424 undergoing rebalancing, proxyGetAll() would be called with an
425 empty list even if rebalancing wasn't happening
427 Release 0.70 on 1/27/2010
429 Changes made since 0.60.1:
431 * A beta of rebalancing (dynamic cluster expansion) support merged
432 into the main branch. See the project's wiki for more information:
433 http://wiki.github.com/voldemort/voldemort/voldemort-rebalancing
434 * New failure detector merged into the main branch:
435 http://wiki.github.com/voldemort/voldemort/failure-detection
436 * Beta mechanism for restoring all of node's data from replicas on
437 demand. This is an alternative to a more gradual mechanism provided
438 by read-repair: useful when a machine is down for a prolonged period
439 and is then re-inserted into the cluster.
440 Invoked via JMX: the operation is restoreDataFromReplication in the
441 voldemort.server.VoldemortServer MBean, with a mandatory parameter
442 (integer >= 1) indicating the number of transfers to do in parallel.
443 * Simple gossip protocol (for cluster metadata) merged
444 into the main branch. Disabled by default: use "enable.gossip=true"
445 to enable, use "gossip.interval.ms" to set an interval at which gossip
446 occurs (default: 30000 i.e., 30 seconds).
447 * Fixed issue 190: add a way of aggregating performance data over
449 * Fixed issue 181: stack trackes shouldn't be filled for Obsolete
452 Release 0.60.1 on 12/18/2009
454 Minor changes made since 0.60:
456 * Better logging in the exception thrown if config/.temp and
457 config/.version are copied
458 * Bumping up the version to 0.60.1 in order to release updated
459 archives, fixing an error in the stores.xml for single_node_cluster
462 Release 0.60 on 12/15/2009
464 Changes made since 0.57.1:
466 * Admin Client/Server API: adds support for streaming-based transfer
467 of entries between nodes, deleting entries on remote nodes,
468 remotely deleting and updating metadata
469 * EC2 testing: a way to periodically run integration and performance
470 tests which involve Voldemort instances on different machines
471 * Experimental support for views
472 * Interpolation search for read-only stores
473 * Support for large lists and strings in the JSON serializer
474 * LZF compression support
475 * Ruby client contributed
476 * Fixed issue 170: hanging if a port is used by another process
477 * Fixed issue 122: suspicious integer division in
478 RequestCounter.getThroughput
479 * Miscellaneous improvements and bug fixes for read-only stores
480 * Fixed issue 168: added StorageEngine.keys()
482 Release 0.57.1 on 11/27/2009
484 Minor change made since 0.57:
486 * Modified build.xml to exclude .git directory from release tarballs/zipfiles
488 Release 0.57 on 11/16/2009
490 The following changes were made since 0.56:
492 * Fixed an issue in ReadOnlyEngine's close() method
493 * Fixed hidden logging in StorageService
494 * Fix for issue 163 (lock mode during get)
495 * Exposed bdb environment stats with setFast(false)
497 Release 0.56 on 10/26/2009
499 The following changes were made since 0.55:
501 * Fix for issue 164: Changed default bdb.max.logfile.size to 60MB
502 * Make file deletes asynchronous in read only store swap
503 * Added better debug logs for bdb stats
504 * Fixed race condition in AbstractSocketPoolTest
505 * Added improved monitoring for bdb stats
506 * Added backoff and retry logic in bootstrap code
507 * Not logging obosleteVersionException(s) or counting it in JMX exception count
509 * C++ client building on OS X
510 * Rely only on the number of versions returned to decide whether to retrieve
511 the value for put(K,V)
512 * Implemented issue 152: getVersion() API in Store
514 Release 0.55 on 10/7/2009
516 The following changes were made since 0.52 (in summary):
518 * Add an event throttler
519 * Added DataCleanupJob
520 * Protocol buffers at version 2.2.0
521 * BDB JE upgraded to 3.3.87
522 * Added data compression support (enable by adding
526 to value-serializer or key-serializer sections in stores.xml)
527 * Added a resource pool/socket pool implementation (no longer using
529 * Added server-side NIO support (enabled by setting
530 enable.nio.connector=true
531 in server.properties)
532 * Improved the efficiency of the protocol buffers network protocol by using
533 CodedOutputStream/CodedInputStream (zero copy transfers)
534 * Fix for issue #21: incorrectness in vector clock inconsistency resolver
535 * Upgrade google-collections
536 * Support for building read only stores in Hadoop