I ran into an interesting problem recently. A de-duplicated pool containing TDP for Oracle backups was consuming much more space than would otherwise be indicated. Here’s what the occupancy looked like:
Node Name Storage Number of Logical
Pool Name Files Space
Occupied
(MB)
---------- ---------- ----------- -----------
CERN_ORA_ADMIN CERNERDISK 810 31,600.95
CERN_ORA_BUILD CERNERDISK 1,189 74,594.84
CERN_ORA_CERT CERNERDISK 402 3,876,363.50
CERN_ORA_TEST CERNERDISK 905 7,658,362.00
LAW_ORA_PROD CERNERDISK 1,424 544,896.19
OEM_ORA_RAM CERNERDISK 2,186 524,795.31
That works out to about 12.7 TB. And, here’s what the storage usage looked like:
Storage Device Estimated Pct Pct High Low
Pool Name Class Name Capacity Util Migr Mig Mig
Pct Pct
----------- ---------- ---------- ----- ----- ---- ---
CERNERDISK CERNERDISK 47,319 G 90.4 90.4 97 90
That’s about 47TB of storage, 90% used, which works out to just over 42TB of used storage. On top of that, TSM was reporting a “savings” of about 2TB, which means I should have about 44TB of data stored on disk. But only 12.7TB was actually backed up!
IBM has built a few interesting scripts to collect TSM data for support lately. One of which is tsm_dedup_stats.pl. This little Perl script collects quite a bit of information relating to deduplication. Here’s some summary info from that script ran a couple of days later:
Pool: CERNERDISK Type: PRIMARY Est. Cap. (MB): 48445474.5 Pct Util: 88.7 Reclaim Thresh: 60 Reclaim Procs: 8 Next Pool: FILEPOOL Identify Procs: 4 Dedup Saved(MB): 2851277 Logical stored (MB): 9921898.18 Dedup Not Stored (MB): 2851277.87 Total Managed (MB): 12773176.05 Volume count: 4713 AVG volume size(MB): 9646 Number of chunks: 847334486 Avg chunk size: 87388
There’s some interesting stuff in there. There’s almost 10TB of logical storage in the storage pool, almost 3TB saved in deduplication, and about 12TB total managed storage, which matches the output from “Q OCCupancy” pretty closely. The output also has a breakdown by client type and storage pool of the deduplication rate:
Client Node Information
-----------------------
DP Oracle: 7
Stats for Storage Pool: CERNERDISK
Dedup Pct: 22.28%
TDPO: 1
Stats for Storage Pool: CERNERDISK
Dedup Pct: 23.14%
So far so good, tsm_dedup_stats.pl matches what we’re seeing with the regular TSM administrative commands.
At this point, I ran “REPAIR OCC“. There’s a possible issue where the occupancy reported and the storage reported by “Q STG” can be inacurate. This new command validates and corrects the numbers reported. Unfortunately, running this had no effect on the problem.
The next thing we looked at was the running deduplication worker threads. After the “IDentify DUPlicates” command locates and marks “chunks” as duplicates, background processes run and actually remove the duplicated chunks. Running “SHOW DEDUPDELETE”, one of the undocumented show commands in TSM, reports the number of worker threads defined, the number of active threads, and which node and filesystem IDs are currently being worked on. If all the worker threads are active for a significant amount of time, more worker threads can be started by putting the “DEDUPDELETIONTHREADS” option in the dsmserv.opt file and restarting the server. The default is 8, on the bigger servers I’ve bumped that to 12. Bumping this number will generate more log and database traffic as well as drive more CPU usage, so you’ll want to keep an eye on that.
I only had 4 threads busy routinely, so adding more threads wouldn’t have helped. But, those threads had were always working on the same node and filespace. The node IDs and node names can be pulled out of the database by running this as the instance owner:
db2 connect to tsmdb1 db2 set schema tsmdb1 db2 "select NODENAME, NODEID, PLATFORM from Nodes"
Those 4 node IDs mapped to 4 of the nodes with data in our problem storage pool. You can see how much work is queued up per node ID with this SQL:
db2 connect to tsmdb1 db2 set schema tsmdb1 db2 "select count(*) as \"chunks\", nodeid from tsmdb1.bf_queued_chunks group by nodeid for read only with ur"
What was happening is that clients were backing up data faster than the TSM server could remove the duplicate data. Part of the problem is probably that so much data is concentrated in one filespace on the TDP clients, so only one deduplication worker thread can process each nodes data at any one time.
We could do client-side deduplication to take the load off the server, but we’ve found that with the big TDP backups that slows down the backup too much. So, with only saving about 2TB of storage on 12TB of backed up data, we came to the conclusion that just turning off deduplication for this storage pool was probably our best bet. After turning off deduplicaiton, it took about 10 days to work through the old duplicate chunks. Now the space used and occupancy reported are practically identical.