Add support for new variant of Columbus dataset

[sbesson]

While working on #1, @dominikl reported failures while applying the rendering settings:

Traceback (most recent call last):
 File "/opt/omero/server/OMERO.server/bin/omero", line 130, in <module>
   rv = omero.cli.argv()
 File "/opt/omero/server/OMERO.server/lib/python/omero/cli.py", line 1626, in argv
   cli.invoke(args[1:])
 File "/opt/omero/server/OMERO.server/lib/python/omero/cli.py", line 1101, in invoke
   stop = self.onecmd(line, previous_args)
 File "/opt/omero/server/OMERO.server/lib/python/omero/cli.py", line 1178, in onecmd
   self.execute(line, previous_args)
 File "/opt/omero/server/OMERO.server/lib/python/omero/cli.py", line 1260, in execute
   args.func(args)
 File "/home/dlindner/omero-cli-render/src/omero_cli_render.py", line 333, in _wrapper
   return func(self, *args, **kwargs)
 File "/home/dlindner/omero-cli-render/src/omero_cli_render.py", line 644, in set
   imgchannels = img.getChannels()
 File "/opt/omero/server/OMERO.server/lib/python/omero/gateway/__init__.py", line 8484, in getChannels
   if not self._prepareRenderingEngine():
 File "/opt/omero/server/OMERO.server/lib/python/omero/gateway/__init__.py", line 7985, in _prepareRenderingEngine
   self._re = self._prepareRE(rdid=rdid)
 File "/opt/omero/server/OMERO.server/lib/python/omero/gateway/__init__.py", line 7959, in _prepareRE
   re.resetDefaultSettings(True, ctx)
 File "/opt/omero/server/OMERO.server/lib/python/omero/gateway/__init__.py", line 4691, in __call__
   return self.handle_exception(e, *args, **kwargs)
 File "/opt/omero/server/OMERO.server/lib/python/omero/gateway/__init__.py", line 4688, in __call__
   return self.f(*args, **kwargs)
 File "/opt/omero/server/OMERO.server/lib/python/omero_api_RenderingEngine_ice.py", line 2326, in resetDefaultSettings
   return _M_omero.api.RenderingEngine._op_resetDefaultSettings.invoke(self, ((save, ), _ctx))
omero.InternalException: exception ::omero::InternalException
{
   serverStackTrace = ome.conditions.InternalException:  Wrapped Exception: (java.lang.RuntimeException):
loci.formats.FormatException: Invalid image number: 4 (series=0, imageCount=4)
	at ome.io.bioformats.BfPixelsWrapper.getPlane(BfPixelsWrapper.java:265)
	at ome.io.bioformats.BfPixelBuffer.getPlane(BfPixelBuffer.java:209)
	at ome.services.RawPixelsBean.findMinMax(RawPixelsBean.java:762)
	at ome.logic.RenderingSettingsImpl.computeLocationStats(RenderingSettingsImpl.java:927)
	at ome.logic.RenderingSettingsImpl.resetChannelBindings(RenderingSettingsImpl.java:856)
	at ome.logic.RenderingSettingsImpl.resetDefaults(RenderingSettingsImpl.java:608)
	at ome.logic.RenderingSettingsImpl.resetDefaults(RenderingSettingsImpl.java:524)
	at ome.logic.RenderingSettingsImpl.resetDefaults(RenderingSettingsImpl.java:1429)
	at ome.services.RenderingBean$11.doWork(RenderingBean.java:2196)
	at sun.reflect.GeneratedMethodAccessor314.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:573)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at com.sun.proxy.$Proxy72.doWork(Unknown Source)
	at ome.services.util.Executor$Impl.execute(Executor.java:452)
	at ome.services.util.Executor$Impl.execute(Executor.java:396)
	at ome.services.RenderingBean._resetDefaults(RenderingBean.java:2193)
	at ome.services.RenderingBean.internalReset(RenderingBean.java:713)
	at ome.services.RenderingBean.resetDefaultSettings(RenderingBean.java:677)
	at sun.reflect.GeneratedMethodAccessor1577.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at com.sun.proxy.$Proxy97.resetDefaultSettings(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor1577.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at com.sun.proxy.$Proxy97.resetDefaultSettings(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor1579.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172)
	at ome.services.throttling.Callback.run(Callback.java:56)
	at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
	at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140)
	at ome.services.blitz.impl.RenderingEngineI.resetDefaultSettings_async(RenderingEngineI.java:380)
	at sun.reflect.GeneratedMethodAccessor1578.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at omero.cmd.CallContext.invoke(CallContext.java:85)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at com.sun.proxy.$Proxy99.resetDefaultSettings_async(Unknown Source)
	at omero.api._RenderingEngineTie.resetDefaultSettings_async(_RenderingEngineTie.java:316)
	at omero.api._RenderingEngineDisp.___resetDefaultSettings(_RenderingEngineDisp.java:2017)
	at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:2437)
	at IceInternal.Incoming.invoke(Incoming.java:221)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
	at Ice.ConnectionI.message(ConnectionI.java:1056)
	at IceInternal.ThreadPool.run(ThreadPool.java:395)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
	at java.lang.Thread.run(Thread.java:748)

   serverExceptionClass = ome.conditions.InternalException
   message =  Wrapped Exception: (java.lang.RuntimeException):
loci.formats.FormatException: Invalid image number: 4 (series=0, imageCount=4)
}

Looking at the stack trace server-side, the error is thrown while trying to access individual planes:

Caused by: loci.formats.FormatException: Invalid image number: 104 (series=0, imageCount=4)
	at loci.formats.FormatTools.checkPlaneNumber(FormatTools.java:1008) ~[formats-api.jar:0.5.3]
	at loci.formats.FormatTools.checkPlaneParameters(FormatTools.java:996) ~[formats-api.jar:0.5.3]
	at loci.formats.in.MinimalTiffReader.openBytes(MinimalTiffReader.java:283) ~[formats-bsd.jar:0.5.3]
	at loci.formats.in.ColumbusReader.openBytes(ColumbusReader.java:198) ~[formats-gpl.jar:0.5.3]
	at loci.formats.ImageReader.openBytes(ImageReader.java:466) ~[formats-api.jar:0.5.3]
...

The reader allowing to read data exported from a PerkinElmer Columbus system was introduced specifically in the context of two IDR studies: id0019 and idr0020. As can be seen from the description or representative samples, the reader currently expects one file containing the pixel data (TIFF) per well samples.

In the case of idr0056, the dataset is structured such that all the well samples within one well are stored within a FLEX file. In order to be able to read the original submitted data, the reader needs to be updated to support this variant of the file format.

Sample plate: /uod/idr/filesets/idr0056-stojic-lncrnas/20190513-ftp/RNAi\ screenA/1921\[652\]/

[melissalinkert]

Looking into this a bit, I can't reproduce plane opening issues with the sample plate. A simple iteration over every plane:

import loci.formats.ImageReader;

public class Test {
  public static void main(String[] args) throws Exception {
    try (ImageReader reader = new ImageReader()) {
      reader.setId(args[0]);
      for (int s=0; s<reader.getSeriesCount(); s++) {
        reader.setSeries(s);
        for (int p=0; p<reader.getImageCount(); p++) {
          /* debug */ System.out.println("opening plane " + p + " from series " + s);
          reader.openBytes(p);
        }
      }
    }
  }
}

opens 4 planes per series across all 9240 series without an exception. Is the sample plate the same as the one that was imported and is throwing exceptions?

Assuming not, my initial guess is that there is at least one truncated file, which we've encountered in other formats but don't thoroughly handle in ColumbusReader. ome/bioformats#3380 adds an extra check and relevant logging (which probably isn't a bad idea in any case), so that's at least something to try.

[melissalinkert]

Sorry, realized that the last comment reflects testing with the wrong branch (https://github.com/openmicroscopy/bioformats/commits/develop). With https://github.com/idr/bioformats/commits/master, I can reproduce the original exception. I think IDR/bioformats#11 will fix it, but ome/bioformats#3380 is probably still not a bad idea to have eventually.

[sbesson]

Thanks @melissalinkert. I confirmed your statement by configuring one of the representative plates failing with IDR Bio-Formats and having it testing using the development Bio-Formats branch - see https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-test-folder/5390/console. I assume the issue above should have failed the pixels test.

From ome/bioformats@v5.9.2...IDR:IDR-0.6.6, it looks like there was no upstream changes in ColumbusReader as this reader was backported almost identically to the mainline Bio-Formats. My suspicion is that the issue above comes from the TIFF reading stack and was incidentally fixed during the Bio-Formats 6 work.

Thanks for opening both issues. Immediately, I will working on trying to merge the Bio-Formats 6 work into IDR Bio-Formats which hopefully should bring the fixes for this dataset. If this is not sufficient or the upgrade work ends up being much harder than initially planned, we can probably come back to IDR/bioformats#11 for a fix specific to this study.