Non absolute path for active:fls?

Poster Content
nk4um User
Posts: 168
May 30, 2011 20:02

Yes, this works! Thanks a lot for your help! That's a small step for mankind but quite a leap for me. :)

Like · Post Reply
nk4um Administrator
Posts: 607
May 30, 2011 18:57

I think you will still need to set the no-cache header on the fls request otherwise that will cache regardless of what you do in this calling endpoint.

def myRequest = context.createRequest("active:fls")
def myFLSConfig =  myRequest.addArgument("operator", "res:/annonces/fichiers/configuration-fls");
myRequest.setHeader("no-cache",true);
def myResponse = context.issueRequest(myRequest);

You are attaching the golden thread to the response from this endpoint not from fls. That is correct. You can look at the response from your endpoint in the visualizer to confirm it has an expiry which has a dependency on the golden thread.

You're nearly there!

Tony

Like · Post Reply
nk4um User
Posts: 168
May 30, 2011 17:35How can we visualize Golden Threads?

Hi Tony!

I think I have attached and cut the Golden Thread at the right places, but for some reason, the FLS output is not refreshed. Is there a way to check the existing Golden Threads? The Visualizer tells me that the attachGoldenThread response is null. Is that ok?

The call to FLS is here:

    def myRequest = context.createRequest("active:fls")
    def myFLSConfig =  myRequest.addArgument("operator","res:/annonces/fichiers/configuration-fls");
    def myResponse = context.issueRequest(myRequest);

    def myGoldenThreadRequest = context.createRequest("active:attachGoldenThread");
    myGoldenThreadRequest.addArgument("id", "urn:fr:annonces:fichiers");
    context.issueRequest(myGoldenThreadRequest);

    context.createResponseFrom(myResponse);

Is that what you meant with "attaching a Golden Thread to FLS"?

While I'm at it, this is how I call "cut":

        def myRequest = context.createRequest("active:cutGoldenThread");
        myRequest.addArgument("id", "urn:fr:annonces:fichiers");
        context.issueRequest(myRequest);

Grégoire

Like · Post Reply
nk4um User
Posts: 168
May 30, 2011 13:19

Hi Tony!

Exactly, I won't use XRL2 because it would have been used to substitute FLS. But since FLS is usable, there's no need to use XRL2 and multiple golden threads (except for performance with lots of files, but I'm not there yet!).

Thanks! Grégoire

Like · Post Reply
nk4um Administrator
Posts: 607
May 30, 2011 13:06

Ok I understand. Yes attaching the golden thread to the output of fls is probably the most direct. However I would suggest the simplest solution is to just have one golden thread that represents the whole directory. When any file is changed or added you invalidate it.

I'm a little unclear as to how you are using xrl2. Maybe that isn't needed if you only have one golden thread?

Cheers, Tony

Like · Post Reply
nk4um User
Posts: 168
May 26, 2011 21:33

Hi Tony,

Yes, I'm trying to do it with Golden Threads. Attaching a Golden Thread with the fls operation sounds like a good idea (I thought that I couldn't use fls at all and that I tried to recode the functionality with Groovy).

What I want to do is this: I've got a large number of files in a directory. Each file contains the real estate adverts for a given town (so there could be theorically thousands of files to scan). Files can be created (NEW), read (SOURCE) and updated (SINK). What I want is to get the fls representation of the directory, but when new files are added, the representation must be rebuilt.

So my original idea was to have one Golden Thread for each file, and to use XRL2 to generate an fls-like representation of the directory, just like in the golden-thread demo. But I was wondering how I could get the list of existing Golden Threads to generate the XRL2 template.

Maybe your idea of attaching a Golden Thread to fls itself is more realistic, as I won't have the problem of listing the numerous golden threads which already exist. The drawback is that, when using a single Golden Thread, each new file in the directory will force the regeneration of the fls representation.

Grégoire

Like · Post Reply
nk4um Administrator
Posts: 607
May 26, 2011 15:51

Hi Gregoire,

there are two broad ways you can solve this:

1) using golden threads. This the way you describe above. In this approach all updates to the filesystem resources must go through an accessor/accessors that cut a golden thread. And then the fls operation must attach a golden thread.

2) write a custom expiry function to attach to the fls response which looks for filesystem changes. This approach will work even if files are externally modified.

Can I first confirm that you definitely want to explore the first option? (If so then we need to start talking about how to do this and how many golden threads you need. One thread will be the simplest approach but I need to understand your requirements better)

Cheers, Tony

Like · Post Reply
nk4um User
Posts: 168
May 26, 2011 15:43

test

Like · Post Reply
nk4um User
Posts: 168
May 26, 2011 06:16

Hi Tony!

Any option that would perform better that would work if the filesystem is only being updated through NetKernel services is to look into using golden threads.

I'm working on this since a few days now, and I'd like to have your opinion about the way I'm using (or trying to use) Golden Threads.

If I understood your suggestion correctly, I need an accessor to deal with the files and take care of the associated Golden Threads. So, I've created an accessor (extending StandardAccessorImpl) with onSource, onSink and onNew methods:

  • On the onSource and onNew, I call active:attachGoldenThread (no problem here)
  • On the onSink, I first call active:cutGoldenThread, then active:attachGoldenThread.

The idea behind the onSink is to tell NetKernel that the resource associated with the virtual resource has been updated, but also to make sure that there is always a virtual ressource associated with each file (and this is where I'm not sure to be clear with the Golden Threads concept).

My problem is with the service that uses this accessor to simulate the "active:fls" output. Is there a way I can get a list of all the existing virtual resources in my module? This would avoid me a time-consuming "scan directory" operation that would only be used to "learn" the virtual ressources id (based on the files names, obviously).

Any advice would be appreciated!

Cheers, Gregoire

Like · Post Reply
nk4um Administrator
Posts: 607
May 7, 2011 20:54Custom expiry function
You can attach custom expiration functions to a response when you develop an accessor. INKFResponse.setExpiry(...) It might make sense for your usage to create such a function that might look for changes and only invalidate when changes are detected. However scanning many files/directories in java is quite slow too. So you might want to make the expiration function smarter so that it only checks at most every so many seconds.

Any option that would perform better that would work if the filesystem is only being updated through NetKernel services is to look into using golden threads.

Cheers,
Tony
Like · Post Reply
nk4um User
Posts: 168
May 7, 2011 19:11"No cache" is slowing down my system
Hi Tony!

Quote:

Just looked at the code and yes it appears that fls does not expire the response. Try adding a NO_CACHE header to the request and see if that gives the required semantics.

See:
http://localhost:1060/book/view/book:guide:physicalreference/doc:physicalreference:request:headers

Cheers,
Tony


I finished building my "places+counters" resource. While it works fine, it is annoyingly slow. The reason is that the lowest-level accessor is FLS (listing the files on the disk is the first operation needed for generating this resource). Hence all the intermediate representations are recomputed each time.

If FLS cannot detect if a directory has changed (or not) since the last call, is there another way to list the files in a directory that would not have this problem?

Thanks!

Gregoire

PS: thanks Gary for the feedback. I''ll try that when my problem with FLS will be fixed.
Like · Post Reply
nk4um User
Posts: 92
May 6, 2011 22:05That worked
Thanks Gregoire, that worked.

It worked for me with some modifications.

import org.netkernel.module.standard.StandardSpace
import org.netkernel.layer0.nkf.INKFRequestContext


URI myRessourceURI = getModuleURI(context);
String root = "<root>"+myRessourceURI.toString()+"</root>";
response = context.createResponseFrom(root);



private URI getModuleURI(INKFRequestContext aContext)
{
    def myModule = null;
    def myGroovyModule = aContext.getKernelContext().getOwningSpace().getOwningModule();
    def myResultModule = null;
    def myScope = aContext.getKernelContext().getRequestScope();
    while ((myScope!=null) && (myResultModule==null)) {
      def mySpace = myScope.getSpace();
      if (mySpace instanceof StandardSpace) {
        myModule = myScope.getSpace().getOwningModule();
        if (myModule != myGroovyModule) {
          myResultModule = myModule;
        }
      }
      myScope = myScope.getParent();
    }
    return myResultModule.getSource();
}


Along with a stylesheet to buld my fls syntax.

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:nk="http://netkernel.org" version="1.0">
  <xsl:outputmethod="xml" />
  <xsl:templatematch="/">
    <fls>
      <root>
        <xsl:value-ofselect="root" />/resources/
      </root>
      <filter>.*txt</filter>
      <recursive />
      <uri />
    </fls>
  </xsl:template>
</xsl:stylesheet>


This gives me a directory structure for my resources, where I look for documentation and build out a Docs.xml and Books.xml dynamically based on the directory structure.
Like · Post Reply
nk4um User
Posts: 168
May 6, 2011 20:05
Hi Gary,

Tony once gave me the following Groovy function that makes possible to find a file which is inside a module''s root directory:

    import org.netkernel.http.transport.HttpAccessorImpl;
    import org.netkernel.layer0.nkf.INKFRequestContext;
    import org.netkernel.module.standard.StandardSpace;
    import org.netkernel.module.standard.StandardModule;
    import java.net.URI;

    private URI getModuleURI(INKFRequestContext aContext)
    {
        def myModule = null;
        def myGroovyModule = aContext.getKernelContext().getOwningSpace().getOwningModule();
        def myResultModule = null;
        def myScope = aContext.getKernelContext().getRequestScope();
        while ((myScope!=null) && (myResultModule==null)) {
          def mySpace = myScope.getSpace();
          if (mySpace instanceof StandardSpace) {
            myModule = myScope.getSpace().getOwningModule();
            if (myModule != myGroovyModule) {
              myResultModule = myModule;
            }
          }
          myScope = myScope.getParent();
        }
        return myResultModule.getSource();
    }

    public void onPost(INKFRequestContext context) throws Exception
    {
        URI myRessourceURI = getModuleURI(context).resolve("relative/path/to/a/file.xml");
        File myFile = new File(myRessourceURI);
    }


I think this function could be used to dynamically generate the arguments requiring absolute paths.

Hope this helps!

Gregoire
Like · Post Reply
nk4um User
Posts: 92
May 6, 2011 18:32Determining location
Hi Tony,

Getting the install path for me does not work because the module may or may not exist under the NK installation.

When using the below code, there is no method for getOwningModule - at least in my installation.  I checked that I have what I think is the latest version.  If there is a jar file that I might be missing or version let me know.  My layer0 module version is 1.65.57.
URI moduleDir=context.getKernelContext().getOwningSpace().getOwningModule().getSource();

In the docs for the sls, it indicates a future of having a filter option.  Has the future arrived?

Thanks
Gary

Like · Post Reply
nk4um Administrator
Posts: 607
May 6, 2011 07:16sls
Also consider the space ls accessor in ext:system:

http://localhost:1060/book/view/book:mod:reference/doc:ext:system:sls

Cheers,
Tony
Like · Post Reply
nk4um Administrator
Posts: 607
May 6, 2011 07:14Determining location on filesystem
Hi Gary, you''ll need to do some string concatenation in XSLT or XRL but basically you have two options to get the filesystem location you need:

1)source netkernel:/config/netkernel.install.path

2) create an accessor and use some code like this to determine the path of the current module:

URI moduleDir=context.getKernelContext().getOwningSpace().getOwningModule().getSource();

Cheers,
Tony
Like · Post Reply
nk4um User
Posts: 92
May 5, 2011 21:36Relative Path?
I did not quite understand your comment Tony on how to get to the relative path by using XRL or XSLT.  How do I get to the base path.  I have a resources folder within my NK project and it will be deployed on many environments.   How do I use fls -root to indicate to start in this resources folder without specifying the absolute path?
Like · Post Reply
nk4um Administrator
Posts: 607
May 4, 2011 18:58
The effect of request and response headers is independent of whether an endpoint is private or not.

I''m not sure why you observed correct behavior when you set a response header. I would not expect this to work. The only way for this to work properly is to set a request header. The reason is he request to fls has already been executed and cached before the mapper endpoint (which is setting the response header) returns it''s response.

Cheers, Tony
Like · Post Reply
nk4um User
Posts: 168
May 4, 2011 15:59
Ok I think I now see the difference between a request and a response "no-cache" header.

When my endpoint is public and when I access it directly, the response header is taken into account, as I wrote earlier.

But ultimately, this endpoint will be private (and without the http transport), the resource being imported and called from another endpoint. In this case, the "no-cache" header seems to be ignored (the returned representation is out of date).

Gregoire
Like · Post Reply
nk4um User
Posts: 168
May 4, 2011 14:46
I don''t understand. The "res:/liste-annonces-vente" resource isn''t cached when this header is set, and so the active:fls accessor is recalled each time, giving me an up-to-date representation of the folder. Basically this is what I want. Why would it be wrong?

Gregoire
Like · Post Reply
nk4um Administrator
Posts: 607
May 4, 2011 14:31
Setting the header on the endpoint inside the mapper only allows setting response headers. Currently there is no way to set request headers using the mapper.

You''ll need to create an endpoint and use NKF to set a request header.

Alternatively fls could be enhanced to support not caching the response based on a configuration value. Is that something you could do? (The source is included in all the modules)

Cheers,
Tony
Like · Post Reply
nk4um User
Posts: 168
May 4, 2011 14:28Ok, found out
Hi,

Just tried with lowercase (name="no-cache") and it works.

Thanks!
Gregoire
Like · Post Reply
nk4um User
Posts: 168
May 4, 2011 14:21Setting no-cache header?
Hi,

So I''m trying with :

<endpoint>
  <grammar>res:/liste-annonces-vente</grammar>
  <request>
    <identifier>active:fls</identifier>
    <argumentname="operator">
      <fls>
        <root>file:/home/greg/dev/NetKernel/modules/urn.fr.services.annonces/resources/data/vente/</root>
        <filter>.*xml</filter>
      </fls>
    </argument>
  </request>
  <headername="NO_CACHE">NO_CACHE</header>
</endpoint>


but it only works after the module is redeployed (if I change the "module.xml" file).

Gregoire
Like · Post Reply
nk4um Administrator
Posts: 607
May 4, 2011 12:50
Just looked at the code and yes it appears that fls does not expire the response. Try adding a NO_CACHE header to the request and see if that gives the required semantics.

See:
http://localhost:1060/book/view/book:guide:physicalreference/doc:physicalreference:request:headers

Cheers,
Tony
Like · Post Reply
nk4um User
Posts: 168
May 4, 2011 11:05
Hi Tony

Thanks! I understand your point of view.

I think I have another problem: my fls endpoint always return the cached representation, and so it doesn''t detect the changes in the directory, i.e. new or deleted files. Is this the expected behaviour?

Gregoire
Like · Post Reply
nk4um Administrator
Posts: 607
May 4, 2011 08:02Composite resources
Hi Gregoire,

this is a common problem with a well defined solution. We took the approach that in general it is better not to overload the functionality of anyone accessor with possible features that may be needed. It is better to allow accessors to be combined to create the functionality needed. (This is similar to the pipelining of command line tools in unix)

So in this case the operator document should be templated and filled in with values using something like XRL of XSLT.

Cheers,
Tony
Like · Post Reply
nk4um User
Posts: 168
May 4, 2011 07:25Non absolute path for active:fls?
Hi!

Is there a way to avoid the use of an absolute path for the "root" parameter of "active:fls"?

At the moment I have:

<config>
  <endpoint>
    <grammar>res:/annonces-
      <groupname="typeAnnonce">
        <regex>(vente|location)</regex>
      </group>
    </grammar>
    <request>
      <identifier>active:fls</identifier>
      <argumentname="operator">
        <fls>
          <root>file:/home/greg/dev/NetKernel/modules/urn.fr.services.annonces/resources/data/vente/</root>
          <filter>.*xml</filter>
        </fls>
      </argument>
    </request>
  </endpoint>
</config>


and I would prefer not to use absolute paths for obvious reasons. Is it possible?

As a side note, I already had the same problem with "active:exec" working-directory parameter (wd), and hadn''t solved it yet.

Thanks for any help!

Gregoire
Like · Post Reply