modules loading modules

Poster Content
pjr
nk4um Moderator
Posts: 755
July 13, 2010 19:19InitEndpoint
Jeff,

Modify the org.netkernel.ext.system.init.InitEndpoint in the ext-system module. Notice it is a listener on the ModuleManager and starts the boot up steps when the first phase of the ModuleManager starting the stem system is over.

I think the reason you saw errors calling mm.sync() was cos your module was being booted by the ModuleManager so you were messing with it as it was starting up.

The InitEndpoint shows how to hook in at postCommission(), wait for stable state, then do your app level boot up.

P.
pjr
nk4um Moderator
Posts: 755
July 13, 2010 19:09ModuleManager
OK - now I understand better.  Actually the ModuleManager really does do what you wanted!

So here''s what you can do.  You''re basically proposing an application-level equivalent of the Init endpoint.  Like Unix, thing that runs in the stem system is the postCommission() in the Init endpoint located in the ext-system module.

Init reads the modules.xml and adds them into the ModuleManager - it then steps the system up to the specified runlevel - which triggers commissioning modules etc.

All you need to do is to decide your application set''s runlevel (make it something high like 10 or 20) and add your modules as a set at that level - then set the runlevel...



private static int APPLICATION_RUNLEVEL=20;

public class ModLoader extends StandardAccessorImpl {
    public void postCommission(INKFRequestContext context) throws Exception {
        ModuleManager mm=ModuleManager.getSingleton();
        mm.addModule(new URI("file:/u/nk/mod-1/"), APPLICATION_RUNLEVEL);
        mm.addModule(new URI("file:/u/nk/mod-2/"), APPLICATION_RUNLEVEL);
        //mm.sync(true);  //Don''t do this.
        mm.setRunLevel(APPLICATION_RUNLEVEL);  //Starts your app set.
    }


Incidentally if you want to stop your set of application modules (but not deregister them - ie "pause" them)...

mm.setRunLevel(APPLICATION_RUNLEVEL-1);  //Stop your app set.

For sure, the ModuleManager has no persistence, in fact it doesn''t know anything about modules.xml.  But your User-side Init will always load your stuff up each time it is commissioned - ie each time NK boots.

If you wanted you could abstract this module loading code into and accessor and simply give it a reference to a my-apps-modules.xml and could start and stop your modules programmatically too.  I guess you see the picture.

Does this help?

P.
jrogers
nk4um User
Posts: 101
July 13, 2010 18:47
My goal is to load groups of related functionality together, but the group of functionality is bigger than a single module (or at least, bigger than I want to maintain in a single module.xml file; separating the space definitions into separate files would provide pretty  much the exact same result).  We''ll most likely use a preprocesed modules.xml file, I was just investigating if there are other ways to go about it (since everything is supposed to be dynamic, modules are just another resource, and there''s a tantalizingly named "ModuleManager" singleton that one might be led to assume manages modules, except it doesn''t).

The issue with not writing to modules.xml is not a governance issue, it''s deployment methodology.  Applications are deployed to a filesystem they have no write access to; writable files are kept elsewhere.  While it has some downsides it does protect against certain catastrophic bugs (like a typo deleting the entire filesystem.)
pjr
nk4um Moderator
Posts: 755
July 13, 2010 10:44
modules.xml is the single point of control for loading all modules.  Maybe I''m not understanding your use case - you want to dynamically set up a set of application modules?  Or do you want to statically set up pre-installed set of modules? Either way you list them in modules.xml (first case is dynamic and could be achieved with code above, 2nd is just static config with no "writing of modules.xml" at runtime).

I guess the real missing info here is that I don''t understand your governance policy requirements.

One possibility is to educate your IT team about dynamic module loading but have all your modules signed and encrypted - so nothing runs that is not signed and no code or resources are visible to anyone but the development team (even the IT people can''t inspect the code).  FYI NKEE has this capability.

In this latter case you could have dynamic editing of modules.xml to point to your application set but know that nothing could be run unless it was locked down.  You could even then avoid writing your own management tools and use apposite to point at your trusted locked down repository of signed / encrypted modules (using ARP to create and manage the repo).

P.

PS There is no definitive pronunciation for apposite but I favour the first one here http://dictionary.reference.com/browse/apposite .  Incidentally apposite is a synonym for ''apt'' which is the name of the Debian package model.  Pertinently appropriate,  you might even say apposite.
jrogers
nk4um User
Posts: 101
July 12, 2010 21:37
There is no need to do anything after this.  modules.xml is monitored for changes and as soon as you sink the updated copy the module differences will be incorporated by the ModuleManager.


Will this still work if modules.xml isn''t writable?  The problem with apposite isn''t that we can''t agree on how to pronounce it (app-o-sit, app-o-site,  or a-pos-it ???), it''s that it writes modules.xml, which we can''t do in our deployment environment.

-J
pjr
nk4um Moderator
Posts: 755
July 12, 2010 21:22Edit modules.xml
Hi Jeff,

Apposite is just a higher-level application above modules.xml - if you like its a glorified editor for modules.xml.

The way it works is to source etc/modules.xml as IHDSNode - then iterates building a new HDS wtih HDSBuilder (copy nodes over and taking care of deduping etc for apposite''s needs).

You could do the same thing but have a application set resource etc/FindlawModules.xml - and just merge this into the HDSBuilder.

The example code for apposite is in .../proc/updateModulesXML.gy


curset=context.source("active:appositeModulesSource", IHDSNode.class);

//Find removed set
b=new HDSBuilder();
icurset=HDSFactory.toHDSInversion(curset);
oldset.getNodes("/resultset/row").each{
   src=it.getFirstValue("LOCALSRC");
   if(icurset.getNodes(src)==null)
   {   b.addNode("removed", src);
   }
}
removedset=b.getRoot();

devset=context.source("active:appositeModulesDeveloper", IHDSNode.class);

//Create modules.xml list
maxrunlevel=1;
b=new HDSBuilder();
b.pushNode("modules");
curset.getNodes("/resultset/row").each{
   b.pushNode("module",  it.getFirstValue("LOCALSRC"));
   rl=it.getFirstValue("RUNLEVEL");
   if(rl>maxrunlevel)
   {   maxrunlevel=rl;      
   }
   b.addNode("@runlevel", rl);
   b.popNode();
}
//Merge in existing developer modules so long as not in removed set
iremovedset=HDSFactory.toHDSInversion(removedset);
devset.getNodes("/modules/module").each{ m->
   src=m.getValue();
   if(iremovedset.getNodes(src)==null)
   {   //merge this module in...
      b.importNode(m);
   }
}
b.popNode();

modulesxml=context.source("res:/apposite/locationof/modules.xml");
//First backup existing
modulesbak=modulesxml+".previous";
prev=context.source(modulesxml, IReadableBinaryStreamRepresentation.class);
context.sink(modulesbak, prev);


There is no need to do anything after this.  modules.xml is monitored for changes and as soon as you sink the updated copy the module differences will be incorporated by the ModuleManager.

Peter

PS We''d be very reticent to change the "KISS" format of modules.xml as it would be against the composite Unix-like design we aim for with the toolset.  As it is we can have raw system set-up, apposite and any number of potential user/system apps all able to work with the same resource.
jrogers
nk4um User
Posts: 101
July 12, 2010 20:47modules loading modules
I''m looking for a simple way to create groups of modules and load them all together.  Ideally, I''d like to be able to load a set of modules by adding a single entry to modules.xml.  I''ve come up with a few ideas, none really satisfying.

* preprocessed modules.xml - maintain a template of modules.xml to be processed by some macro/inclusion facility (e.g., cpp or m4) to produce a complete modules.xml file. 


* globbing in modules.xml i.e.,
<modulerunlevel="7">/u/nk-root/modules/app-group-1/*</module>

Doesn''t work, unlikely to change.


* subgrouping in modules.xml -
this doesn''t meet my "single line" criteria, but does make the modules.xml file a little more readable.
<app-group>
  <id>my-app-group</id>
  <module>my/module/1</module>
  <module>my/module/2</module> ...
</app-group>

This works (since it appears that a <module> element is found anywhere in the file, not just a child of the root element), but it relies on undocumented behavior (i.e., subject to change) and it gets mangled by automatic updates (apposite).


* module that loads other modules.  For this I tried to create a module that would load other modules in its postCommission phase:
public class ModLoader extends StandardAccessorImpl {
    public void postCommission(INKFRequestContext context) throws Exception {
        ModuleManager mm=ModuleManager.getSingleton();
        mm.addModule(new URI("file:/u/nk/mod-1/"));
        mm.addModule(new URI("file:/u/nk/mod-2/"));
        mm.sync(true);
    }

This almost works - it loads the modules, but the mm.sync() call throws an exception;  and the next time there is a hot restart all the sub-modules are unloaded.


Right now the preprocessed modules.xml is looking like our best option.  But I''m wondering if you have any alternate suggestions.  Apposite isn''t an option for us because we''re deploying to a read-only filesystem (i.e., modules.xml can''t be updated at runtime).

Thanks
-J