#404 ✓closed
Ben Hood

Bulk difference report

Reported by Ben Hood | November 1st, 2011 @ 02:19 PM | in 1.0 (closed)

In addition to the sequential escalation callbacks that are currently supported, it would be good to be able to give a participant a bulk report to summarize the differences that have arisen from a scan. This would be convenient for scenarios where large volumes of differences are expected to occur. This would allow a participant to perform a bulk operation that in certain circumstances would be more efficient than trickling repairs out by having the agent request them sequentially. The scope of this ticket involves:

  • Configuring a participant to accept bulk difference reports as a consequence of running a scan (it should also be triggerable manually);
  • Add the ability to accept a report submission as a separate participant trait (as opposed to adding this to endpoints);
  • Defining a report as a child of a pair element in the XML and that report element would have a POST URL in it;
  • Defining a report format that permits streamed reading and writing, e.g. CSV;
  • Putting in the plumbing to send and receive streams via HTTP posts (maybe also with gzip compression);

Comments and changes to this ticket

  • Ben Hood

    Ben Hood November 2nd, 2011 @ 01:56 AM

  • Ben Hood

    Ben Hood November 2nd, 2011 @ 02:51 AM

  • Ben Hood

    Ben Hood November 2nd, 2011 @ 05:43 AM

  • Ben Hood

    Ben Hood November 2nd, 2011 @ 06:37 AM

  • Ben Hood

    Ben Hood November 2nd, 2011 @ 09:47 PM

  • Ben Hood

    Ben Hood November 3rd, 2011 @ 02:16 AM

  • Ben Hood

    Ben Hood November 3rd, 2011 @ 02:40 AM

  • Ben Hood

    Ben Hood November 3rd, 2011 @ 03:31 AM

  • Ben Hood

    Ben Hood November 3rd, 2011 @ 03:31 AM

  • Paul Jones

    Paul Jones November 3rd, 2011 @ 03:33 AM

    • Assigned user changed from “Paul Jones” to “Ben Hood”
    • State changed from “new” to “resolved”

    Pull request raised and ready for QA.

  • Ben Hood

    Ben Hood November 3rd, 2011 @ 04:45 AM

    • Assigned user changed from “Ben Hood” to “Paul Jones”
    • State changed from “resolved” to “open”

    When running the participant demo, I'm getting the following error in the agent:

    04:43:49.860 [qtp85964998-56] INFO Configuration:153 - DiffaPairRef(WEB-1,diffa) -> Processing pair declare/update request ....
04:43:49.889 [qtp85964998-56] INFO PairActorSupervisor:83 - Stopped actor for key: diffa/WEB-1
04:43:49.891 [qtp85964998-56] INFO Configuration:158 - DiffaPairRef(WEB-1,diffa) -> Stopping pair actor  (PT0.011S)
04:43:50.286 [qtp85964998-56] INFO PairActorSupervisor:66 - Started actor for key: diffa/WEB-1
04:43:50.286 [qtp85964998-56] INFO Configuration:163 - DiffaPairRef(WEB-1,diffa) -> Starting pair actor  (PT0.345S)
04:43:50.287 [qtp85964998-56] INFO Configuration:164 - DiffaPairRef(WEB-1,diffa) -> Adding to matching manager (PT0S)
04:43:50.288 [qtp85964998-56] INFO Configuration:165 - DiffaPairRef(WEB-1,diffa) -> Adding to differences store (PT0S)
04:43:50.289 [qtp85964998-56] INFO Configuration:166 - DiffaPairRef(WEB-1,diffa) -> Registering with scheduler (PT0S)
04:43:50.289 [qtp85964998-56] INFO Configuration:167 - DiffaPairRef(WEB-1,diffa) -> Running initial difference (PT0S)
04:43:50.290 [qtp85964998-56] INFO Configuration:170 - DiffaPairRef(WEB-1,diffa) -> Completed pair declare/update request
04:43:50.290 [qtp85964998-56] INFO Configuration:153 - DiffaPairRef(WEB-2,diffa) -> Processing pair declare/update request ....
04:43:50.302 [qtp85964998-56] INFO PairActorSupervisor:83 - Stopped actor for key: diffa/WEB-2
04:43:50.302 [qtp85964998-56] INFO Configuration:158 - DiffaPairRef(WEB-2,diffa) -> Stopping pair actor  (PT0S)
04:43:50.619 [qtp85964998-56] INFO PairActorSupervisor:66 - Started actor for key: diffa/WEB-2
04:43:50.619 [qtp85964998-56] INFO Configuration:163 - DiffaPairRef(WEB-2,diffa) -> Starting pair actor  (PT0.273S)
04:43:50.620 [qtp85964998-56] INFO Configuration:164 - DiffaPairRef(WEB-2,diffa) -> Adding to matching manager (PT0.001S)
04:43:50.620 [qtp85964998-56] INFO Configuration:165 - DiffaPairRef(WEB-2,diffa) -> Adding to differences store (PT0S)
04:43:50.620 [qtp85964998-56] INFO Configuration:166 - DiffaPairRef(WEB-2,diffa) -> Registering with scheduler (PT0S)
04:43:50.621 [qtp85964998-56] INFO Configuration:167 - DiffaPairRef(WEB-2,diffa) -> Running initial difference (PT0S)
04:43:50.621 [qtp85964998-56] INFO Configuration:170 - DiffaPairRef(WEB-2,diffa) -> Completed pair declare/update request
04:43:50.772 [qtp85964998-56] ERROR ApplicationSpecificExceptionMapper:32 - Unhandled application exception
org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
    at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:96) ~[hibernate-core-3.6.7-2L.jar:3.6.7-2L]
    at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) ~[hibernate-core-3.6.7-2L.jar:3.6.7-2L]
    at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:275) ~[hibernate-core-3.6.7-2L.jar:3.6.7-2L]
    at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:268) ~[hibernate-core-3.6.7-2L.jar:3.6.7-2L]
    at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:184) ~[hibernate-core-3.6.7-2L.jar:3.6.7-2L]
    at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) ~[hibernate-core-3.6.7-2L.jar:3.6.7-2L]
    at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51) ~[hibernate-core-3.6.7-2L.jar:3.6.7-2L]
    at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216) ~[hibernate-core-3.6.7-2L.jar:3.6.7-2L]
    at net.lshift.diffa.kernel.util.SessionHelper.withSession(SessionHelper.scala:33) ~[diffa-kernel-0.9-SNAPSHOT.jar:na]
    at net.lshift.diffa.kernel.config.HibernateDomainConfigStore.createOrUpdateEscalation(HibernateDomainConfigStore.scala:130) ~[diffa-kernel-0.9-SNAPSHOT.jar:na]
    at net.lshift.diffa.kernel.frontend.Configuration.createOrUpdateEscalation(Configuration.scala:250) ~[diffa-kernel-0.9-SNAPSHOT.jar:na]
    at net.lshift.diffa.kernel.frontend.Configuration$$anonfun$applyConfiguration$10.apply(Configuration.scala:76) ~[diffa-kernel-0.9-SNAPSHOT.jar:na]
    at net.lshift.diffa.kernel.frontend.Configuration$$anonfun$applyConfiguration$10.apply(Configuration.scala:76) ~[diffa-kernel-0.9-SNAPSHOT.jar:na]
    at scala.collection.immutable.Set$Set1.foreach(Set.scala:86) ~[scala-library-2.9.0-1.jar:na]
    at net.lshift.diffa.kernel.frontend.Configuration.applyConfiguration(Configuration.scala:76) ~[diffa-kernel-0.9-SNAPSHOT.jar:na]
    at net.lshift.diffa.agent.rest.ConfigurationResource.applyConfiguration(ConfigurationResource.scala:45) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_26]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_26]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_26]
    at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_26]
    at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$VoidOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:167) ~[jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) ~[jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134) ~[jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) [jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) [jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) [jersey-server-1.8.jar:1.8]
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) [jersey-server-1.8.jar:1.8]
    at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) [jersey-server-1.8.jar:1.8]
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) [jersey-server-1.8.jar:1.8]
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) [jersey-server-1.8.jar:1.8]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) [servlet-api-2.5.jar:2.5]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:534) [jetty-servlet-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1352) [jetty-servlet-7.3.0.v20110203.jar:7.3.0.v20110203]
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129) [sitemesh-2.4.2.jar:na]
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77) [sitemesh-2.4.2.jar:na]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1323) [jetty-servlet-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:177) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168) [spring-security-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) [spring-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) [spring-web-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1323) [jetty-servlet-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:474) [jetty-servlet-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:517) [jetty-security-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:935) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:404) [jetty-servlet-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:184) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:870) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:247) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:151) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.Server.handle(Server.java:346) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1068) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:807) [jetty-http-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:220) [jetty-http-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) [jetty-server-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:520) [jetty-io-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) [jetty-io-7.3.0.v20110203.jar:7.3.0.v20110203]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:528) [jetty-util-7.3.0.v20110203.jar:7.3.0.v20110203]
    at java.lang.Thread.run(Thread.java:680) [na:1.6.0_26]
Caused by: java.sql.BatchUpdateException: integrity constraint violation: NOT NULL check constraint; SYS_CT_10065 table: ESCALATIONS column: ORIGIN
    at org.hsqldb.jdbc.JDBCPreparedStatement.executeBatch(Unknown Source) ~[hsqldb-2.2.4.jar:2.2.4]
    at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297) ~[commons-dbcp-1.2.2.jar:1.2.2]
    at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70) ~[hibernate-core-3.6.7-2L.jar:3.6.7-2L]
    at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268) ~[hibernate-core-3.6.7-2L.jar:3.6.7-2L]
    ... 84 common frames omitted
  • Ben Hood

    Ben Hood November 3rd, 2011 @ 05:06 AM

    (from [05f11e67e76f761a8a2b7c0c68046cab8169ff18]) [#404] Resolve QA notes:

  • Paul Jones

    Paul Jones November 3rd, 2011 @ 05:07 AM

    That is actually a bug in the configuration of HSQLDB - it isn't putting database files in target/, but instead creating a set of files called mem.*. This means that schema changes that don't have a migration (eg, WIP ones) don't get applied.

    I've reconfigured the agent database setup to use the target directory.

  • Ben Hood

    Ben Hood November 3rd, 2011 @ 05:10 AM

  • Ben Hood

    Ben Hood November 3rd, 2011 @ 05:22 AM

    Are you sure that the mem.* is not a derivative artifact of HSQLDB? According to http://hsqldb.org/doc/guide/ch01.html#N101CA, this is how memory-only DBs are created.

  • Paul Jones

    Paul Jones November 3rd, 2011 @ 05:24 AM

    According to those docs, nothing is written to disk - but this file (if you look at the .script) is a complete representation of the whole state. In the unit tests, I don't see anything appear on disk - but for some reason, the agent itests configured via JNDI keep creating this file. I don't think the agent having a persistent store in integration is a big deal - as long as its in target (which I've added a change to do).

  • Ben Hood

    Ben Hood November 3rd, 2011 @ 05:25 AM

    OK, so I've been able to see the report in the participant web's log file - nice touch :-)

  • Paul Jones

    Paul Jones November 3rd, 2011 @ 06:52 AM

    • Assigned user changed from “Paul Jones” to “Ben Hood”
    • State changed from “open” to “resolved”

    All outstanding QA issues have been addressed now. Ready for second pass QA.

  • Ben Hood

    Ben Hood November 3rd, 2011 @ 11:27 AM

    • State changed from “resolved” to “closed”

    QA+

  • Ben Hood

    Ben Hood November 5th, 2011 @ 11:35 AM

    • Milestone set to “1.0”
    • Milestone order changed from “139” to “0”

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

A Real Time Differencing Tool.

Shared Ticket Bins

People watching this ticket

Tags

Referenced by

Pages