Datastore Conversion

TiG
@tig
7 years ago
184 posts

What is the recommended method to make bulk modifications to a data store?

In our case, we need to (occasionally) change the value of one property (a count) and in some cases add a new (derived) property. This will be done on the jrComment datastore so in our case the conversion will process an enormous number of items.

Logically the conversion algorithm will work on the comments for one discussion, group discussion or blog at a time. Thus it could take place when either of these aggregates are accessed. That is, it could be on-demand rather than converting the entire store all at once.

Bottom line, just looking for the JR prescribed method for datastore conversions.

--
TiG
updated by @tig: 03/30/18 04:44:28AM

@michael
7 years ago
7,746 posts

There is the "Batch Item" update module that allows you to update multiple items from the ACP. Install it and each module that uses a datastore will have an extra tool in their TOOLS tab.

https://www.jamroom.net/the-jamroom-network/documentation/modules/2922/batch-item-editor

If you're after a function, then jrCore_db_update_multiple_items() is probably what you're after.

TiG
@tig
7 years ago
184 posts

@michael

We need to do an algorithmic conversion. The Batch editor appears to be designed for making manual changes.

Recently Paul and Brian made an update to jrComment which introduced concatenated keys for performance. I am looking to follow their lead and use the same method (although in my case I am not doing this to introduce concatenated keys). So I would have code that would inspect the comments and make the selective adjustments as needed.

I presume they did a db backup, used some mechanism to invoke their code to create the new concatenated keys and possibly some mechanism to verify integrity. I could roll my own method but I figured it best to do what JR does.

--
TiG
updated by @tig: 12/27/17 10:50:11AM

@michael
7 years ago
7,746 posts

Any database adjustments or updates usually fire off of the integrity check listeners
ACP -> MODULES -> CORE -> SYSTEM CORE -> TOOLS -> INTEGRITY CHECK

So probably the 'repair_module' listener.

Then in there whatever adjustments that need to be done are done. Example, the Gallery Module needed to make sure that all datastore items have a 'gallery_order' value set, so does this in the repair module listener:


function jrGallery_repair_module_listener($_data, $_user, $_conf, $_args, $event)
{
    $_rt = jrCore_db_get_items_missing_key('jrGallery', 'gallery_order');
    if ($_rt && is_array($_rt)) {
        $_up = array();
        foreach ($_rt as $id) {
            $_up[$id] = array('gallery_order' => 100);
        }
        if (count($_up) > 0) {
            jrCore_db_update_multiple_items('jrGallery', $_up);
            jrCore_logger('INF', "updated " . count($_up) . " gallery images missing gallery_order key");
        }
    }
    return $_data;
}

* get all gallery items missing the 'gallery_order' key.
* set that items 'gallery_order' key to 100
* update all those items.

@michael
7 years ago
7,746 posts

AH! This is an important feature for you to know about @tig, queue workers. They are a way to move the processing of whatever needs done to the background so the person running the integrity check (or whatever) doesnt need to wait for the process that was started to finish.

In the performance improvement for jrComment the key is added byt the jrComment_verify_db_worker().

That worker is started in the 'repair_module' listener with:

        // Verify comments
        jrCore_queue_create('jrComment', 'verify_db', array('count' => $num), 0, null, 1);

So if its going to take a long time to complete, pass it off to a queue worker.

Docs: "The Queue System"
https://www.jamroom.net/the-jamroom-network/documentation/module-developer-guide/1543/the-queue-system

TiG
@tig
7 years ago
184 posts

@michael

Perfect! This is clearly the way to go. Thanks much Michael.

--
TiG

Datastore Conversion

Tags