TSM Deduplication Increases Storage Usage (for some values of deduplication)

I ran into an interesting problem recently. A de-duplicated pool containing TDP for Oracle backups was consuming much more space than would otherwise be indicated. Here’s what the occupancy looked like:

Node Name         Storage         Number of     Logical
                  Pool Name           Files     Space
                                                Occupied
                                                (MB)
----------        ----------    -----------    ----------- 
CERN_ORA_ADMIN    CERNERDISK            810      31,600.95 
CERN_ORA_BUILD    CERNERDISK          1,189      74,594.84 
CERN_ORA_CERT     CERNERDISK            402   3,876,363.50 
CERN_ORA_TEST     CERNERDISK            905   7,658,362.00
LAW_ORA_PROD      CERNERDISK          1,424     544,896.19 
OEM_ORA_RAM       CERNERDISK          2,186     524,795.31

That works out to about 12.7 TB. And, here’s what the storage usage looked like:

Storage         Device          Estimated       Pct       Pct     High  Low  
Pool Name       Class Name       Capacity      Util      Migr      Mig  Mig  
                                                                   Pct  Pct  
-----------     ----------     ----------     -----     -----     ----  ---  
CERNERDISK      CERNERDISK       47,319 G      90.4      90.4       97  90 

That’s about 47TB of storage, 90% used, which works out to just over 42TB of used storage. On top of that, TSM was reporting a “savings” of about 2TB, which means I should have about 44TB of data stored on disk. But only 12.7TB was actually backed up!

IBM has built a few interesting scripts to collect TSM data for support lately. One of which is tsm_dedup_stats.pl. This little Perl script collects quite a bit of information relating to deduplication. Here’s some summary info from that script ran a couple of days later:

Pool: CERNERDISK
  Type: PRIMARY		   Est. Cap. (MB): 48445474.5  Pct Util: 88.7
  Reclaim Thresh: 60	Reclaim Procs: 8		  Next Pool: FILEPOOL
  Identify Procs: 4	  Dedup Saved(MB): 2851277


  Logical stored (MB):	  9921898.18
  Dedup Not Stored (MB):  2851277.87
  Total Managed (MB):	 12773176.05

  Volume count:			        4713
  AVG volume size(MB):	        9646
  Number of chunks:	       847334486
  Avg chunk size:	           87388

There’s some interesting stuff in there. There’s almost 10TB of logical storage in the storage pool, almost 3TB saved in deduplication, and about 12TB total managed storage, which matches the output from “Q OCCupancy” pretty closely. The output also has a breakdown by client type and storage pool of the deduplication rate:

Client Node Information
-----------------------
    DP Oracle:		7
      Stats for Storage Pool:		CERNERDISK
        Dedup Pct:		22.28%
    TDPO:		1
      Stats for Storage Pool:		CERNERDISK
        Dedup Pct:		23.14%

So far so good, tsm_dedup_stats.pl matches what we’re seeing with the regular TSM administrative commands.

At this point, I ran “REPAIR OCC“. There’s a possible issue where the occupancy reported and the storage reported by “Q STG” can be inacurate. This new command validates and corrects the numbers reported. Unfortunately, running this had no effect on the problem.

The next thing we looked at was the running deduplication worker threads. After the “IDentify DUPlicates” command locates and marks “chunks” as duplicates, background processes run and actually remove the duplicated chunks. Running “SHOW DEDUPDELETE”, one of the undocumented show commands in TSM, reports the number of worker threads defined, the number of active threads, and which node and filesystem IDs are currently being worked on. If all the worker threads are active for a significant amount of time, more worker threads can be started by putting the “DEDUPDELETIONTHREADS” option in the dsmserv.opt file and restarting the server. The default is 8, on the bigger servers I’ve bumped that to 12. Bumping this number will generate more log and database traffic as well as drive more CPU usage, so you’ll want to keep an eye on that.

I only had 4 threads busy routinely, so adding more threads wouldn’t have helped. But, those threads had were always working on the same node and filespace. The node IDs and node names can be pulled out of the database by running this as the instance owner:

db2 connect to tsmdb1
db2 set schema tsmdb1
db2 "select NODENAME, NODEID, PLATFORM from Nodes"

Those 4 node IDs mapped to 4 of the nodes with data in our problem storage pool. You can see how much work is queued up per node ID with this SQL:

db2 connect to tsmdb1
db2 set schema tsmdb1
db2 "select count(*) as \"chunks\", nodeid from tsmdb1.bf_queued_chunks group by nodeid for read only with ur"

What was happening is that clients were backing up data faster than the TSM server could remove the duplicate data. Part of the problem is probably that so much data is concentrated in one filespace on the TDP clients, so only one deduplication worker thread can process each nodes data at any one time.

We could do client-side deduplication to take the load off the server, but we’ve found that with the big TDP backups that slows down the backup too much. So, with only saving about 2TB of storage on 12TB of backed up data, we came to the conclusion that just turning off deduplication for this storage pool was probably our best bet. After turning off deduplicaiton, it took about 10 days to work through the old duplicate chunks. Now the space used and occupancy reported are practically identical.

Leave a Reply

Your email address will not be published. Required fields are marked *