[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: fast delete a lot of entries

Derek Zhou wrote:
> Karsten
> 
> Unfortunately, down time is even worse than slow. 
> 
> I wish OpenLDAP has an integrated search and delete operation, like delete entries based on a search filter, all executed in the same transaction in the server. That should be more efficient than doing search and delete from the client side because the B+ trees are just traversed once. Also the server can  impose size limit on how many entries can be deleted in one go, just like the search, to avoid someone hogging the server. 
> 
> For the LDAP gurus here, the above idea should be technically possible, right?

In normal operation, as opposed to simple experimentation, why do you actually need to delete
a large number of entries quickly? If you just need to make them unavailable to a bunch of
clients, you can first do a subtree rename to move them out of the way, and then delete
incrementally at whatever speed.

> 
> Derek
>  
> On Monday, November 26, 2018 03:36:16 PM Karsten Heymann wrote:
>> Hi Derek,
>>
>> no idea about the deletion speed, but if a short downtime is an option
>> and the entries to be deleted can be matched by a search filter, you
>> could do a
>>
>>   slapcat -a '(!(<filter of what should be deleted>))'
>>
>> and the slapadd the resulting file into an empty database. Depending
>> on the amount of to entries be deleted in comparison to the total
>> number of entries in the database, this could be quicker than live
>> deletes.
>>
>> best regards
>> Karsten
>> Am Sa., 24. Nov. 2018 um 18:56 Uhr schrieb Derek Zhou <derek@shannon-data.com>:
>>>
>>> dear list,
>>>
>>> I was experimenting with slapd by adding a lot of entries then deleting them. What I found is adding speed is not bad but deletion speed is lacking. With or without dbsync, delete speed is around half of that of adding. Naively, I thought deleting should be easier than adding, because in adding you actually need to pass and write whole entry of data, while in delete you can just mark the database page as free? The DB file did not shrink after massive deletion; which kind of suggest that deleting is only marking page as free, not really return them to the OS. I am using the latest git tree and the mdb backend.
>>>
>>> Another related idea is delete a whole branch from the DIT. LDAP is already hierarchical, to delete all entries under a branch, one would assume that there must be an better way than deleting entries with a client side script, like what I am doing? With SQL you can drop a table. With LDAP, can I delete a whole branch?
>>>
>>> --
>>> Derek Zhou
>>> Shannon Systems
>>> http://www.shannon-sys.com
>>>
> 


-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/