August, 2005 -- August, 2006
Dbdb a JDI-based Platform for
Cross-language Debugging
Table
of contents
Dbdb
a JDI-based Platform for Cross-language Debugging
Related
work and the value of Dbdb
Speaking too soon: false starts and
misdirections
Listing
1. org.hrum.dbdb.OracleExample1
Listing
2. PL/SQL stored procedures
Appendix
C. Criticism of BEAs Patent Application
Appendix
D. Other use cases for single-stack debugging
JDWP (Java Debug Wire Protocol)
Introduction
Source-level debuggers, and GUI for them, have been around for a long time and are indispensable tools for many programmers. As most real-world programs involve more than one language, modern IDEs present a consistent graphical front-end to different debuggers, for instance, the Eclipse IDE (for Java, C, C++, Python, Ruby, etc.), Oracles JDeveloper (for Java and PL/SQL debuggers), the ubiquitous Emacs (for everything but the kitchen sink), etc. However, while ability to debug multiple languages in a single IDE is no doubt useful, integration between the different debuggers (of importance to projects consisting of modules spanning several languages) still presents a problem. A notable enhancement is presented by tools such as SCORE, a debugger supporting Ada and C and allowing to seamlessly transition between languages[1].
A still further improvement would be to allow for single-stack debugging of mixed-language programs. In other words, as program in language A calls a program in language B, the programmer sees a single call stack from A to B, rather than two separate ones. In many modern-day enterprise applications, there are countless instances of code in, say, Java, calling stored procedures in PL/SQL. As one who encounters this situation daily, I believe that a seamless transition of debugging from one language to another and the ability to see the call in a single stack would be of immense value to the multi-language developer. (Other useful applications of this technique are discussed below, in Related work and the value of Dbdb and Other language types chapters.)
To this end, I propose an implementation of a framework for single-stack debugging of multi-language applications based on the JDI (Java Debug Interface), an API layer of JPDA[2] (Java Platform Debugging Architecture).
About this document
In this paper I will present and discuss:
1. An overview of related work in this area, serving as proof that the proposed technique is not merely a whim, but offers a tangible benefit for real-world development
2. The benefits of the proposed solution over the existing related work
3. An overview of the JPDA in general, JDI in particular and the reasons for choosing it as a foundation of the proposed framework, along with a discussion of alternative choices.
4. The working proof of concept, including design, documentation, working code and demo. Here, I will also discuss:
a. the generalization of the proof of concept into a framework
b. directions for further extending the framework, and
c. use cases for plugging into the framework
5. Post-mortem of the project
6. Conclusion and thoughts on further work in this area
More up-to-date
information
This project has been released as Open Source, as more than a few colleagues expressed interest in its ultimate value, and in working on it to achieve such. Thus, a reader is referred to the following up-to-date online maintained sources:
1. Homepage at http://db-db.sourceforge.net[*]
2. Project [b]log at http://fish37.livejournal.com/tag/dbdb
3. Project at Sourceforge.net, at http://sourceforge.net/projects/db-db/
4. DefectBug
tracking at http://sourceforge.net/tracker/?group_id=150446&atid=777815
5. Demo at http://www.viewletcentral.com/vc/viewlet.html?id=87625530
Tools
This project was implemented and tested with the Eclipse 3.2 IDE (www.eclipse.org), Java Standard Edition 6 Mustang (https://mustang.dev.java.net/), and Oracle 10g.
As for the creation of this document, free trial of EndNote (www.endnote.com) was very useful for managing references in this document. http://www.phruby.com/stencildownload.html#Visio2003.
Related work and the value of Dbdb
The value of seamless, single-stack cross-language debugging is recognized in the industry. For example:
1.
Stylus Studio features
an XSL debugger features include the ability to seamlessly
switch contexts and step into Java XSLT extension functions using Stylus
Studio's integrated Java IDE, then returning back to calling the XSLT
stylesheet.[3] (See
also Other language types chapter below).
2. According to Abdul Al-Azzawe, DB2 Development Tools Architect, IBM is investigating cross-language debugging support in the same call stack, such as mixing of Java and SQL nested stored procedure calls[4]. I am not aware of any implementation available to the public to date.
3.
According to David Alpern of Oracle, their combined-VM
algorithms
are more general [
] and from the early stages of this work it was
clear to us that it would be desirable to also support combined-VM
debugging more generally for cross-tier calls of various sorts. However, it is
unclear whether, if at all, this functionality (if it exists) will be
available, as he notes that he is not at this time allowed to say when
cross-tier debugging support might actually appear in our released products.[5]
4.
Engineers at BEA Systems also thought about this
problem, and submitted a patent application, for a method that provides, in
part, that if more that one language
appears on a stack, a developer can see the frames for each language, as well
as be able to inspect variables for each language.[6] This particular offering is examined in more detail in Appendix C.
Criticism of BEAs Patent Application below.
Other
possible uses of single-stack cross-language debugging includes stepping into C
implementations of native Java methods, for example. In
describing a technique debugging these kinds of mixed-language applications, Matthew
White of IBM laments, do you effectively debug the Java/C hybrid
programming since there is no debugger available that can check on this
software chimera?[7]. Almost answering his plea are Eclipse developers, listing
investigat[ing] debugging from Java into natives and back as potential debugger work items and
areas of investigation for 2.1 and future releases of Eclipse[8].
Additionally,
Wrapped Application Debugger (WAD)[9, 10] attempts to address integration of C/C++ code with scripting
language (Perl, Python, Tcl) extensions by embedding a debugger into the
application. The authors complaint that little work has been done on the
area of debugging in mixed-language environment can perhaps be assuaged, at
least in part, by considerations presented here.
A single-stack
view may also be of interest in debugging distributed applications using remote
calls (RPC, RMI, EJB, etc.) Lovas et al. discuss a framework (using JPDA) usage
for a step-into mechanism for remote method invocations (RMI) that unifies the
debugging mechanism for local and remote calls[11].
Further,
a casual perusal of various discussion groups reveals the desire for the
single-stack debugging popping up often enough; some of these is summarized in Appendix D.
Other use cases for single-stack debugging
And, finally, upon taking a quick look at http://db-db.sourceforge.net, Darin Wright of Eclipse Debug team had this to say: it certainly looks interesting. I know that people are interested in cross language debugging in general. [ ] I wonder if it could be applied in a more general way to address cross-language debugging, or if it is specific to stored procedures/SQL and Java?[12]
Why JDI?
First, of course, in order to have a multi-language debugger (never mind combined call stacks yet), one needs a common framework that provides access to the needed debuggees information. I propose using the JDI layer of JPDA (Java Platform Debugger Architecture) as such a framework. As I will show, JDI is really quite language-independent (J for Java notwithstanding)[].
What is JDI?
JPDA consists of three layers of APIs; here, I propose using one of them, JDI (Java Debug Interface). For the discussion of the choice of this layer vs. other JPDA layers, or other similar approaches, see Appendix E.
JDI is a set of Java interfaces, the implementations of which are to be provided by a debuggee that is, the target programs execution environment. At the root of it is the VirtualMachine[]interface which represents the debuggee, that is, the target execution environment. While nominally the debuggee is meant to be a Java Virtual Machine, it does not have to be one. For example, it could be an interpreter, an intermediary program driving yet another debugger (say, gdb), or a native executable specially compiled[§]. As long as the appropriate interfaces are implemented, any JPDA-compliant debugger can control any debuggee. By manipulating these implementations, the debugger is able to query and control the debuggees execution: suspend and resume threads, set breakpoints, evaluate variables and methods, etc. As we will see, these interfaces are generic enough to represent the execution of a program in any imperative programming language.
Two of the methods of VirtualMachine interface, eventRequestManager() and eventQueue(), return implementations of EventRequestManager and EventQueue, respectively. These two objects are the primary channels of communication between the debugger and debuggee. EventRequestManager is used by the debugger to request to be notified when a certain debuggable event occurs. (Such a request can also contain a directive to suspend the thread in which the event occurred). The debugger then polls EventQueue for the events it is interested in. Examples of such events are MethodEntryEvent, ThreadStartEvent, StepEvent, ExceptionEvent, etc.
Each event contains references to other artifacts of the debuggee. For example, as can be expected, a MethodEntryEvent contains a reference to the method entered, and to a thread in which this entry occurred. The thread is represented by ThreadReference interface, which further contains a reference to the stack of StackFrames in this thread. In turn, a StackFrame, which represents the state of one method invocation on a thread's call stack, has references to the variables visible in this frames scope.
Notice the lack of anything Java-specific[**]. JDI, in fact, is so well designed that (especially with the adoption of JSR-45[14]) the J for Java in JPDA is an unnecessary qualifier. Its set of abstractions is similar to that of Eclipses language-neutral debugging framework[] (centering on threads, stack frames, breakpoints and stepping into/over/out).
Finally, JDI is also used very creatively for such purposes as: dynamic analysis of running applications[15], self-healing applications[16], load-time transformation of compiled Java programs for the purposes of Aspect-Oriented Programming[17], dynamic creation of sequence and dependency diagrams from running programs[18], etc.
A rough block diagram of JPDA-debugger integration is shown in Figure 1. The JVMDI and JDWP components of JPDA are discussed in Appendix E. Why not JDI?.
Figure 1. JPDA - debugger block diagram.
Proof of concept
For a proof of concept, I
will consider a simple Java program that calls a simple PL/SQL stored
procedure. The Java program is org.hrum.dbdb.example.OracleExample1. We are interested in the fragment shown in Listing 1. org.hrum.dbdb.OracleExample1 in Appendix A. The stored PL/SQL function FUNC, called from line 25 (and introduced in lines
20-21), and the function it calls in turn (FUNC2) are presented in Listing 2. PL/SQL stored procedures.
Architectural overview
Very roughly, a debugger capable of multiple language support (such as Eclipse) can be described by a block diagram in Figure 2. (Oracle prior to 10 is specified to provide yet another way of interfacing with the actual debugger; 10 and after support JDWP[19]).
Figure 2. Multi-language debugger
The proposed solution would look as shown in Figure 3. The Dbdb JDI implementation will interface to the debugger in the same way as the Java debugging module did. Everything else is hidden from the IDE. All communications between the IDE and the multiple debuggee targets is mediated by the Dbdb layer, which provides a single stack per thread executed, even though different frames of the stack may represent different languages.
Figure 3. Combined call-stack multi-language debugger
A sample sequence diagram of a debugging session with this approach is shown in Figure 4. (Here, Program 1 calls Program 2).
Figure 4. Sequence diagram of combined call-stack debugging.
The actual implementation is more involved than that, such as:
- Since Program 1 and Dbdb JDI Implementation run asynchronously, the sequence of setting a breakpoint right after the method in Program 1 that calls Program 2 is reached is done by other means. For example, if Program 1 is a JPDA-enabled target, this is done by creating a MethodEntryRequest for the appropriate method that will suspend the thread as soon as the method is entered.
- The modification of arguments step may not be necessary if a debugger can be launched separately and can then be attached to the debuggee process.
Infrastructure
The following are the basic building blocks
for a JPDA-based combined call stack debugger.
1. DbdbVirtualMachine[]
Implementing VirtualMachine, this was originally intended to hold the virtual machines encountered throughout the debugging and switch among them. This is not the case (see below). The primarily function of this class is to present itself to the debugger and return DbdbEventQueue and DbdbRequestManagerInvocationHandler. Everything else this class does is there for a reason but is not really important.
2. DbdbRequestManagerInvocationHandler
Serves as an
implementation of RequestManager for the
virtual machines managed by DbdbVirtualMachine. The client
(debuggers) requests are routed through this class, but components of Dbdb
itself call the appropriate RequestManagers of managed
virtual machines directly. More on this below, under Mediators.
3. DbdbVirtualMachine.Registry
This is merely a collection of Maps, that holds mappings such as:
1.
ThreadReference with which a DelegatingThreadReference was
constructed to the DelegatingThreadReference that
contains it
2.
ObjectReference for a java.sql.Statement to an ObjectReference for a java.sql.Connection which
created this Statement
3.
A
from2To Map, which
associates a ThreadReference in one VM to the one in the VM that will be called
from. For example, it will associate a ThreadReference that calls Statement.execute() in Java VM
to the ThreadReference of the
actual stored procedure that this Statement.execute() invokes in
the Oracle VM.
NOTE: This one is specifically
mentioned here, because it will be referred to heavily in the Algorithm chapter below.
4.
Etc. (See also Registry
reform below).
4. EventListener
hierarchy
Classes implementing this interface are called by the DbdbEventQueue to handle MethodEntryEvent and MethodExitEvent appropriately in their process() method. They are expected to be self-registering (registered upon instantiation). More defails are given below.
5. DbdbEventQueue
Implementation of EventQueue, that is responsible for:
- polling EventQueues of all Virtual Machines being debugged
- calling all EventListeners
- deciding which events to return to the caller and which to consume (based on the return value from EventListener.process() method)
- doing more things than it, perhaps, should upon encountering a StepEvent. This piece of logic should be removed from this class in order for Dbdb to be a more flexible framework. As it stands now, and as will be seen in the Algorithm chapter below, this class is responsible for recognizing
More on this below, under Mediators.
6. DelegatingThreadReference
An implementation of ThreadReference that is responsible for exposing a unified call stack to the user. Internally, it holds a stack of ThreadReferences from managed virtual machines. The Dbdb framework (normally, various EventListeners) will push to and pop this stack as needed.
7. FakeStepEvent,
DelegatingStackFrame, DbdbEventSet
Further infrastructure for supporting the DelegatingThreadReference.
8. org.hrum.dbdb.plugin
package
This package contains the Eclipse plug-in for the Dbdb functionality. It includes code for dynamically looking up code for the stored procedures and displaying it in an editor.
Algorithm
The algorithm employed is as follows (we will use the OracleExample1 program for illustration):
1. When a debuggee is launched, the DbdbVirtualMachine is initialized with:
a. VirtualMachine object gotten from the appropriate Connector
b. Currently, the following EventListeners (others can be added, of course):
i.
__java_sql_DriverManagerListener
ii.
__oracle_jdbc_driver_PhysicalConnectionListener
iii.
__java_lang_RuntimeListener
These are explained below.
2.
On receiving MethodExitEvent from DriverManager.getConnection()
method (line 15), __java_sql_DriverManagerListener:
a.
Obtains the reference
to the java.sql.Connection object as the MethodExitEvents returnValue().
b.
Starts a thread with a ListeningConnector (com.sun.jdi.SocketListen, or, for Eclipse, org.eclipse.jdi.internal.connect.SocketListeningConnectorImpl, listening on a port.
c.
Starts a thread that
will execute DBMS_DEBUG_JDWP.CONNECT_TCP[19] on the connection obtained above. This will establish
a debugging session within Oracle.
d.
Waits until the
connection is made, and obtains the Oracles virtual machine from the ListeningConnector.
e. Instantiates a of OracleDbmsMethodEntryEventListener, which will listen for MethodEntryEvents from the Oracles virtual machine. This
listener, during construction, associates (in Registry.from2To, see 3
above) the Java thread[§§] it the connection was created in with a DelegatingThreadReference.WAITING_FOR_THREAD_REFERENCE
constant.
3.
On receiving a MethodExitEvent from Connection.prepareCall() (line
20-21), __oracle_jdbc_driver_PhysicalConnectionListener:
a. Loads the
source code for the stored procedure being prepared from Oracle (a little
earlier than necessary, but why not
)
b.
Associates reference to the returned java.sql.Statement from the MethodExitEvents returnValue() to the
reference to java.sql.Connection which created the statement.
4.
On receiving a MethodEntryEvent
from an Oracle stored procedure, OracleDbmsMethodEntryEventListener:
a.
Finds the ThreadReference
for the Java thread created in its
constructor (see 2.e
above), creates a DelegatingThreadReference for it, and pushes onto it the ThreadReference from the Oracle stored procedure.
This
step also associates the found ThreadReference with the DelegatingThreadReference in the Registry (instead of earlier WAITING_FOR_THREAD_REFERENCE created in 2.e
above)
b. Instantiates an OracleDbmsMethodExitEventListener
c.
Returns, for the
debugger, a FakeStepEvent,
with this DelegatingThreadReference.
5.
On receiving a MethodExitEvent
from an Oracle stored procedure, OracleDbmsMethodExitEventListener
(see
4.b above):
a.
Executes a popThreadReference()
on the DelegatingThreadReference instance it is constructed with
b.
Resumes the thread[***]
that originally called the popped thread.
Mediators[]
All the while, Dbdbs
implementations of EventRequestManager. and EventQueue mediate the interaction between the debugger and the
debuggee as follows:
6. DbdbRequestManagerInvocationHandler, merely, forwards all requests to EventRequestManagers of all managed VMs. The only special case is a call
to createStepRequest(). If
this is received, the Registry.from2To map (see 3
above)is
checked. If the ThreadReference
on which the step is to be created maps to a to ThreadReference object which is not a WAITING_FOR_THREAD_REFERENCE constant, the request is forwarded, of course, to
the VM of the to ThreadReference.
7.
DbdbEventQueue, every time the debugger wishes to retrieve an EventSet
from the VM it thinks it queries, does
the following:
a.
Polls EventQueues of all managed VMs. All obtained EventSets are added to a local q2 variable.
b.
If q2 is empty, just return null[].
c.
Create a new instance
of DbdbEventSet, called toReturn. This will be the EventSet that will be returned to the debugger; its VM is DbdbVirtualMachine.
d.
Removes the EventSet
from the local q2 variable, and, for each Event evt, executes the algorithm shown below, in Figure
4. DbdbEventQueue algorithm, part 1 .
e.
Finally, if toReturn
set is not empty, it is returned. If it is:
i.
If resumed is false, a
resume is called on it.
ii.
A null
is returned[§§§].
Figure 5. DbdbEventQueue algorithm, part 1
Figure 6.
DbdbEventQueue algorithm, part 2.
The resulting combined call
stack is shown in Appendix B. Screenshots .
Using
Dbdb as a framework
While this may not be a full-fledged framework at this point, let us consider two use cases of a developer wishing to extend this approach.
Registry reform
First and foremost, a better solution should be found for DbdbVirtualMachine.Registry, for this to be truly extensible. But this should, perhaps, wait, until the Dbdb applications has been extended further.
Some thought may be given of implementing a simple relational-database-like data structure in a programming language. In Java, for instance, Map is nice, but what if you could have an easy SymmetricMap, or, going further, something resembling an RDBMS table structure (without the complexity of triggers et al.)
Mediator policy
Secondly, then, a policy of using Dbdb implementations of com.sun.jdi.* interfaces (such as VirtualMachine, Event, EventManager, EventRequest, etc.) vs. managed VMs implementations needs to be clarified. In other words, when is a Dbdb implementation returned to the debugger, and when is it an implementation for the managed VM that the client is currently interested in? Currently it is not explicitly defined. It is my opinion that a few more extensions of Dbdb are needed (such as are outlined in use cases below) in order to formulate this policy better.
Use cases
Now, to the more general things
Actions required of a developer using Dbdb in the use cases listed below could be generalized; in other words, made simpler or pluggable. The comments on how to do that are in corresponding footnote for the item.
Use case 1: Repeat the proof-of-concept on another JDBC-compliant
RDBMS
1. Find
out the appropriate implementation of java.sql.Statement for this RDBMS, and
create an EventListener for it, using __oracle_jdbc_driver_PhysicalConnectionListener
as
a template. In particular:
a.
It is advised that this EventListener extend the MethodXEventListener, which would
require for it to be named in the following pattern (as should be obvious):
i.
The name starts with __
(two underscores).
ii.
The rest of the name is
the fully qualified name of the class implementing java.sql.Statement
for this JDBC vendor, with periods
replaced by underscores, and a suffix Listener added at the end.
iii.
Register this EventListener with the DbdbVirtualMachine[****].
b.
Modify
__java_sql_DriverManagerListener to:
i.
Actually parse the
arguments to getConnection()
method(s[]) to see
whether the JDBC URL provided is the one that is handled by the target RDBMS[].
ii.
If so, get the
appropriate com.sun.jdi.VirtualMachine implementation for the debuggee. (If, unlike Oracle,
no direct JPDA-compliant VirtualMachine implementation is available), see Use case 2. Implement
cross-language debugging of other languages below.
c.
Implement the
equivalents of OracleDbmsMethodEntryEventListener and OracleDbmsMethodExitEventListener[§§§§], using the above classes as guidelines.
Use case 2. Implement
cross-language debugging of other languages supporting JPDA
For the sake of example, let us consider a made-up need to debug another Java program called from our Java program. All that needs to be done is an implementation of DbdbProcessDebugger has to be created (the beginnings can be seen in JavaProcessDebugger class). The fully qualified name of this class has to be registered in dbdb_debuggers.properties file which must be on the classpath[*****].
This is how it works.
Use
case 3. Implement cross-language debugging of other languages
For the sake of example, let us consider a made-up need to debug a Perl script called from Java. In this case, in addition to performing the steps from the above use cases, a developer would need to provide the adapters from the target debugging framework[20] to implementations of com.sun.jdi.VirtualMachine and all related artifacts.
Framework or tool?
As can be seen above, while this not a defined framework/API, it is certainly poised to evolve into one.
Post-mortem[]
A postmortem is a useful exercise. Not that I believe that this project is dead, but this concept can and should be applied even to a release. Thus, for this proof-of-concept, here is a post-mortem. It explains some pitfalls, false leads, and reasons for this project taking more than it should.
Leaky abstractons
The following Leaky abstractions[21] certainly took away time and effort:
1. For
all their talk about good OO design and interfaces, Sun shows that they are sinners[]
as well. It is nice to learn that Sun's implementation when manipulating ostensibly
insterfaces, in fact, expects them to be implementation classes inside. For
example, com.sun.jdi.MirrorImpl.validateMirrorOrNull() inside casts things to com.sun.jdi.MirrorImpl rather than com.sun.jdi.Mirror. Ditto for ThreadReference vs ThreadReferenceImpl.
2.
This is not a big deal, but a caveat nonetheless for those
attempting to integrate Dbdb with exiting debuggers. What is one to do with a null EventSet?
What about an empty one? Javadt does not like a null EventSet -- it just does not check
for nulls (which is ok, for a throwaway reference implementation).
But an empty one is not good for Eclipse, because it indiscriminately calls a resume() on it, which is not what we
want. Back to returning null then. But a thought: how many of such little things would
render this "framework" not really a framework (still, certainly,
usable, but just a nagging feeling
)? Or should this all be configurable?
Speaking too soon:
false starts and misdirections
1.
Because of my general fear of having to spend time on
learning Eclipses plug-in architecture, I proposed putting off integration
with this very popular IDE in favor of using simpler tools. But indeed Quis emendabit ipsos
emendatra[§§§§§]. Eclipse should have been
used from the very beginning. This would have saved quite a lot of time and I
would never learn about the leaky abstraction 2, above
On a related note, see the chapter Why
JDI? above for the discussion of choosing JPDA
over Eclipse Debug Framework. Had I started again, I would probably had used
the latter
The idea of using JDWP was judged to be an overkill, and ultimately abandoned[******] (but see also Appendix E. Why not JDI?). The discovery of Oracles support of JPDA-based debugging would have sped things up. But it happened after I spent inordinate amount of time on task that ultimately is not needed researching ways of using existing JDWP implementations. JSR-45 is not needed as well[].
- The idea of loading debugged classes with an alternate class loader, which would substitute real classes (in this case, JDBC drivers and their subsequent artifacts) for their equivalents, and attendant ideas (such as using bytecode modification) did not find its use, despite much time spent on this research. This was due to:
a. Mouthing off about this in proposal without really knowing that much about JPDA
b. It is hard to debug (but see Parallel drill-down chapter below).
However, this may yet find its use. I just have not thought about it again
- I have labored for some time believing that returnValue() feature exists in JPDA. When I realized that it does not, I was forced to produce an elaborate workaround, which was quite complicated and not extremely robust. It is only after the workaround has been created that I found that this feature indeed exists in JPDA starting with Java 6 so add more time to clean up the ugly workaround, to use more elegant solution with returnValue()[].
Further work
The project is released to open-source, and will continue to evolve[§§§§§§] in its current incarnation as an Eclipse plug-in primarily for stored procedure debugging. In this chapter I will outline some other interesting possibilities for this project.
Other
language types
It would be interesting to develop a use for this framework for debugging not just procedural/imperative[*******] languages for which it is created, but also for:
- Logical/constraint languages (such as Prolog, which has been adapted for Eclipse[22]).
Perhaps as a subset of this functionality, or of an application in its own right, could be adaptation of this technique for debugging parsers (such as those based on grammars ANTLR, yacc, etc.)
- Other declarative languages, such as XSLT.
- Cross-cutting concerns/Aspects (I have not even begun to appreciate this, but I thought Id throw it into the mix[]).
As with Prolog implementation, the three areas above may involve some creative redefinition of semantics of certain debugging operations, such as stepping. These problems, as applied to declarative, and an Eclipse-based solution, are presented in detail by Hui Wu et al. in Grammar-Driven Generation of Domain-Specific Language Testing Tools[23].
- Stepping into, C/C++ implementation of Java native methods, or, more generally, into assembly/machine code/byte code (but this is, perhaps, more of a topic for the Parallel drill-down approach, outlined below).
Parallel drill-down
Often, the call stack actually executed is different from the abstract call stack though of by a developer (the latter being the one depicted as a sequence diagram). In this example, it is useful for a developer to think of a call to java.sql.Statement.execute() as logically leading directly into the stored procedure.
For example, from a standpoint of an application developer, it would be good do away with the Oracle implementation, which we currently include in a combined stack shown in Appendix B. Screenshots . In other words, in this particular screenshot, your average application developer is only interested in first (the Java frame containing a Statement.execute() call) and last (in this screenshot, the first and only, but in general, the first PL/SQL frame) stack frames of the shown call stack. On the other hand, for a developer of Oracles JDBC driver, both may be useful.
For a lack of a better term, Id like to coin one: parallel drill-down (OK, so its not a cool term). This would mean creating a framework for debugging where at a point where logical and implementation scenarios diverge, the developer is presented with multiple parallel call stacks to step through. Any path can be selected, and the developer can switch between paths at will; a step in one path must be synchronized with other possible paths. For all my criticism of the BEA patent application (see Appendix C. Criticism of BEAs Patent Application), the authors correctly identified this particular problem when they wrote: For example, it is not uncommon for a developer to see stack information not directly related to the software being debugged when encountering a stack frame for one language, when using a debugger intended for another language. As another example, when using a debugger intended for the Java language, a Java stack will not include the XScript [ ] stack, and can sometimes show the set of Java classes that implement the XScript engine (these are part of the environment, but not the software the developer is working on).[6]
Obviously, there is no limit to splitting these, however, common scenarios can be a well-defined and be a part of the framework, while also providing the developer with a way to define her own. Here, SMAP mechanism of JSR-45 can probably be leveraged.
Another example is a call to by reflection or related things. For instance, debugging things using java.lang.reflect.Proxy mechanism (such as Dbdbs own) is notoriously painful. We usually want logical debugging the developer may not want to see internal JDK infrastructure, but sometimes we may want implementation one. Since reflection is quite common when using generic framework, this may be a nice-to-have feature.
As an example of the usefulness of this feature, consider the screenshot in Appendix B. Screenshots From the point of view of the application developer, the stack frames between the application code calling Statement.execute() and the stored procedure code are extraneous plumbing.
If this feature is implemented, providers of generic frameworks can provide the split scenarios as part of the product, allowing the user of the framework an easier way to separate the debugging of her logic vs. the compound logic of her components and the underlying framework.
This approach would be even more useful to programs that are worse than those using reflection those using byte code manipulation.
Other
features
Other features that are not implemented in proof of concept, but would be useful, include
drop-to-frame functionality, allowing popping a selected stack frame (and all frames above it) from the execution stack and then stepping back into the frame.[24].
Conclusion
Dixi.
Appendix A. Code listings
Listing 1. org.hrum.dbdb.OracleExample1
01: package org.hrum.dbdb.example;
02:
03: import java.sql.CallableStatement;
04: import java.sql.Connection;
05: import java.sql.DriverManager;
06: import java.sql.Types;
07:
08: public class OracleExample1 {
10:
11: public
static void main(String[] args) {
12: try
{
13: Class.forName("oracle.jdbc.driver.OracleDriver");
14: System.out.println("DriverManager.getConnection()");
15: Connection
con = DriverManager.getConnection(
16: "jdbc:oracle:thin:@localhost:1521:dbdb",
17: "sys",
18: "password");
19: System.out.println("con.prepareCall()");
20: CallableStatement
st =
21: con.prepareCall("begin
?:=func(?); end;");
22: st.setInt(2,
1);
23: st.registerOutParameter(1,
Types.INTEGER);
24: System.out.println("Statement.execute()");
25: st.execute();
26: System.out.println(st.getObject(1));
27: } catch (Exception e) {
28: e.printStackTrace(System.out);
29: System.exit(-1);
30: }
31: }
32: }
Listing 2. PL/SQL stored procedures
The PL/SQL stored procedures are created with the following code[]:
create or replace function func2(num
number) return number
is
begin
return num * 3;
end;
/
create or replace function func(ctr
number) return number
is
ret number:=0;
begin
for i in 1..ctr loop
ret := ret + func2(i);
ret := ret + 3;
ret := ret - 3;
end loop;
return ret;
end;
/
alter function func compile debug;
alter function func2 compile debug; [§§§§§§§]
Appendix B. Screenshots 
Appendix C. Criticism of BEAs Patent
Application
Id like to expound more on this particular proposition, as it is a claim for a patent, especially in light of claims that Oracle has a similar patent filing[5]. No implementation has been made available to my knowledge; and the approach itself, as described in the application, does not seem to be novel and non-obvious in November of 2004. At this time both JPDA and Eclipse Debug Framework (discussed in Why JDI? chapter below) were already available, and given that, a remark that creating debugging tools that can be applied to software applied to more than one programming language, and running in the same environment, has proved to be extremely difficult seems particularly disingenuous. But at least they saw a need, and briefly identified (but not addressed) some problems I also discuss below, in the Parallel drill-down chapter below.
Further,
consider this pronouncement One multi-language debugger, JSR 45, can only be
used to debug languages that are easily transformed into Java and then
compiled. Such a statement makes one wonder whether this was merely lost in
translation to the patent attorney, or the claimants misunderstand or, for the
purposes of patent claim, misrepresent, JSR 45. First, it is not a debugger,
but a specification. Second, JSR 45s stated goal is to establish [a]
mechanism [
] by which programs executed under the Java virtual machine but
written in languages other than the Java programming language, can be
debugged with references to the original source[14] (Emphasis mine). This renders their next point a non-sequitur: This
and most other multi-language debuggers won't work with languages such as
XScript that where [sic] the language will be run by an interpreter or the
language can not be mapped directly to Java because, for example, the language
has a different data structure. Perhaps what they are trying to say is that
this is the case when a language cannot be easily mapped to a
procedural/imperative model (like XScript, perhaps), which indeed seems to be
the case with JPDA. However, they provide no proof that their model does...
Appendix D. Other use
cases for single-stack debugging
I'm wondering: do you have ideas on how to support
cross-language debugging for RDT, ie. a (J)Ruby methods that call Java methods
(and the other way around), ie. the StackTrace would contain both Ruby and Java
stack frames. The question is: if the JRuby process is launched with the
Debugger, you'll have a JDT DebugModel, but you'll also want a Ruby DebugModel.
I'm not sure if there's an easy way to combing StackTraces from both
DebugModels (maybe some kind of Delegating DebugModel that collects StackTraces
from JDT and Ruby DebugModels).
http://www.mail-archive.com/rubyeclipse-development@lists.sourceforge.net/msg00042.html
Is there an
easy way to set up integrated debugging between a Java application and PL/SQL?
A reply to a tutorial on using DBMS_DEBUG, Oracles
debugging package at http://www.orablogs.com/shay/archives/001571.html
In my
private fantasy land, I'd be able to run emacs and somehow invoke pdb with
pdbtrack to do source-level debugging of my python code, then automagically
step into gdb when the python calls out to C++ code I've written via
boost::python.
http://mail.python.org/pipermail/c++-sig/2003-June/004315.html
Appendix E. Why not JDI?
The choice of JDI was a bit opportunistic. It did, indeed, strike me as a fairly generic framework; however, its choice was also influenced by my greater familiarity with it (versus Eclipse Debug Framework) and by relative ease of implementation of the proof-of-concept for a JPDA-compliant debugger (versus JDWP, see below). This chapter attempts to examine alternatives to JDI for the implementation
JDWP (Java Debug
Wire Protocol)
The two API layers left unexamined in here are JDWP (Java Debug Wire Protocol), which which defines the format of information and requests transferred between the debugging process and the debugger front end [2] and JVMTI (JVM Tools Interface), which a programming interface [that] provides both a way to inspect the state and to control the execution of applications running in the Java virtual machine[25]. The latter is outside the scope of this paper[********]. But JDWP deserves further examination here.
JDWP provides a packet-based, stateless protocol for communication between a debugger and a debuggee. It can be said that it defines a serialization model for the JDI objects discussed in chapter What is JDI?, above. Another way of saying it is that the Suns JDI API[] is merely a reference implementation, in Java, of JDWP. In this way, it is completely agnostic as to the implementation of a debugger and can be used as a universal debugging protocol. As the JPDA FAQ points out, [t]heoretically JPDA could have only one interface, the Java Debug Wire Protocol (JDWP)[26]. Indeed, there already is a Common LISP[27] and a Ruby[28, 29] implementation of JDWP.
At this time, despite the two above-mentioned (and encouraging) cases, pretty much all JDWP-aware debuggers are those that are fully JPDA-aware; that is, these are Java debuggers, which already use JDI API. If it is recognized that JDWP can be used not just as a Java debugging protocol, but as a generic one, and other debuggers are aware of it, the single-stack implementation could perhaps be pushed down to the layer that is responsible for reading/writing JDWP packets and creating the data structures, rather than on top of the data structures themselves. In fact, as we saw earlier (and made use of in proof of concept), Oracle, as of version 9i, provides for JDWP-based remote debugging of its stored procedures not only Java, but PL/SQL[19]. This is a real-world example of JPDA usage for non-Java languages.
It is true that [w]riting directly to JDWP however is painstaking work, information sent across the wire must be read and written precisely. But implementations for doing that already exist in Java (GNU Classpath[30]), C (Suns own implementation of JDWP), and, as we have seen above, in Common LISP and Ruby. As it is a well-designed protocol, perhaps the next iteration of this project should be rewritten with JDWP in mind.
Both JDWP and JDI approaches can be used, out of the box, for remote debugging, which is very useful.
And, one a final thought. As I mentioned in chapter What is JDI?, when debugging a native executable, it may be simple enough to create Java mediator programs that would drive the third-party debugger (e.g., gdb) and translate their interactions with this debugger into the objects implementing JDI API. Driving the third-party debugger through, perhaps, a command-line interface (CLI) seems straightforward. However, in case of the gdb debugger, it was found that CLI has proven to be highly unreliable and Eclipse has since switched to use machine-oriented text interface[13] which is quickly becoming a de facto standard for integrating debuggers into a variety of environments [31]. But should using CLI with another third-party debugger present a problem, and there is no MI-like solution, perhaps it could be more efficient to devise a way to enhance a compiler so that the executable itself is JDWP-compliant.
Eclipse Debug
Framework
Obviously, well-designed IDEs that provide multi-language debugging have also implemented language-independent debugging frameworks. Perhaps one of the most well-adopted and best-designed of these[] is Eclipse Debug Framework[32] (hereafter, EDF). Because it is language-neutral, one may think it a naturally better candidate for this project. Here, I will examine this proposition. Full disclosure: large portion of the work was already completed using JDI by the time I have more thoroughly familiarized myself with this alternative, which may make me seem biased against it
Pro EDF
- It is intended to be language-independent. Its design, centered around the notion of threads, stack frames, breakpoints and stepping seems similar to JPDA and so seemingly not well-suited for languages that are not procedural/imperative languages (see the Other language types chapter below). However, even those can be accommodated: for example, a working Prolog debugger[22, 33] utilizing EDF creatively redefines the semantics of stepping into and out[33].
- For
Eclipse-based debuggers, implementing single-stack debugging in this layer
(in a way probably very similar to the one described in this paper) is
easier, because no adaptation is needed for a new language to support
JPDA.
Pro JPDA
- EDF is indeed designed for [m]odeling a set of artifacts and actions common to many debuggers, known as the debug model. For example, some common debug artifacts are threads, stack frames, variables and breakpoints; and some common actions are suspending, stepping, resuming and terminating.[32] However, as we saw earlier, a functionally equivalent set artifacts and actions is available via JPDA. The Pragmatics of Java Debugging article also hints at the cross-language potential of this architecture[34].
- For JPDA-compliant non-Eclipse-based debuggers (and there are a number of those, the obvious advantage is that EDF does not need to be re-implemented. Further, because of the definition of JDWP, and existing implementations of it in a number of languages (Java, C, Ruby and LISP), it can be easier integrated with debuggers not written in Java, and allow for remote debugging as an added bonus.
Why not, indeed?
As we have seen above, the choice of JDI was largely expedience and opportunism. This implementation certainly does the job, and JDI can be used in similar applications to great success. But I am not going to rationalize my choices to no end. I believe that the best implementation is JDWP, as it is closer to a universal debugging protocol.
Appendix F.
References
1. SCORE: Multi-Language Debugger.
http://www.info.uni-karlsruhe.de/~andf/documents/scoremld.pdf
2. Java Platform Debugger Architecture. http://java.sun.com/j2se/1.5.0/docs/guide/jpda/
3. Stylus Studio's XSL Debugger http://www.stylusstudio.com/xsl_debugger.html
4. Al-Azzawe,
A. (2004) Abdul Al-Azzawe on development
enhancements in DB2 Universal Database V8.2. IBM developerWorks. http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0409alazzawe/
5. Alpern,
D., Personal communication (e-mail).
2005.
6. Pugh,
W.A. and J.M. Eckels. Patent Application:
System for multi-language debugging, Provisional application No. 60/450,014. 2003
http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.html&r=1&p=1&f=G&l=50&d=PG01&S1=20040230955.PGNR.&OS=dn/20040230955&RS=DN/20040230955
7. White,
M. (2001) Debugging integrated Java and
C/C++ code: Two approaches using JNI. IBM
developerWorks. http://www-128.ibm.com/developerworks/java/library/j-jnidebug/index.html
8. 2.0.1 Candidate Fixes: Possible Work Items. http://dev.eclipse.org/viewcvs/index.cgi/jdt-debug-home/DebugDirections2.1.htm?rev=1.11
9. Beazley,
D.M. An Embedded Error Recovery and
Debugging Mechanism for Scripting Language Extensions in USENIX. 2001 http://db.usenix.org/events/usenix01/full_papers/beazley/beazley_html/index.html
10. Cao,
J. and D.M. Beazley (2005) Embedded
Debugging of C/C++ Plugins and Extension Modules. Technical report TR-2005-07, Department of Computer Science, University
of Chicago. http://www.cs.uchicago.edu/files/tr_authentic/TR-2005-07.pdf
11. Lovas,
R. and V. Sunderam. Debugging of
metacomputing applications. in International
Parallel and Distributed Processing Symposium. 2002. Ft.Lauterdale, Fl http://www.mathcs.emory.edu/harness/pub/general/lovas4.pdf
12. Wright,
D., Personal communication (e-mail).
2006.
13. Leszek,
P. C/C++ development with the Eclipse
Platform: How to use the C/C++ Development Toolkit (CDT). IBM developerWorks. http://www-128.ibm.com/developerworks/opensource/library/os-ecc/
14. Bracha,
G. and A. Ryman. JSR (Java Specification
Request) 45: Debugging Support for Other Languages. 2003
http://www.jcp.org/en/jsr/detail?id=45
15. Gueheneuc,
Y.-G., R. Douence, and N. Jussien (2002) No
Java without Caffeine: A Tool for Dynamic Analysis of Java Programs. Ecole des Mines de Nantes. http://www.emn.fr/x-info/jussien/publications/gueheneuc-RR0207.pdf
16. Savarese,
D.F. (2002) Application, Heal Thyself.
JavaPro. http://www.fawcette.com/javapro/2002_09/magazine/columns/proshop/default_pf.aspx
(login required)
17. Kniesel,
G., P. Costanza, and M. Austermann, JMangler
- A Powerful Back-end for Aspect-oriented Programming, in Aspect-oriented Software Development,
R.Filman, T.Elrod, and S.Clarke, Editors. 2004, Prentice Hall http://www.informatik.uni-bonn.de/~gk/papers/jmanglerChapterPreprint.pdf
18. Loton,
T. (2001) Using The Java Platform
Debugger Architecture. Java
Developer's Journal. http://java.sys-con.com/read/36221.htm
19. Antognini,
C. Debugging PL/SQL and Java Stored
Programs with JPDA. http://www.trivadis.ch/Images/JPDA_tcm17-7136.pdf
20. Foy,
B.D. (2001) Using the Perl Debugger. Dr.Dobb's Journal. http://www.ddj.com/184404744
21. Spolsky,
J. (2002) The Law of Leaky Abstractions. http://www.joelonsoftware.com/articles/LeakyAbstractions.html
22. Kroening,
M. (2005) Eclipse: Adapting and Updating
an IDE. Dr.Dobb's Journal. http://www.ddj.com/184407751
23. Wu,
H., J. Cray, and M. Mernik. Grammar-Driven
Generation of Domain-Specific Language Testing Tools. in OOPSLA. 2005. San Diego http://www.cis.uab.edu/gray/Pubs/ddf.pdf
24. Eclipse 3.1, documentation of
org.eclipse.debug.core.model.IDropToFrame interface (Javadoc). http://help.eclipse.org/help31/index.jsp?topic=/org.eclipse.platform.doc.isv/reference/api/org/eclipse/debug/core/model/IDropToFrame.html
25. JVM Tool Interface http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/jvmti.html
26. Java Platform Debugger Architecture FAQ. http://java.sun.com/products/jpda/faq.html
27. Lichteblau,
D. CL-JDI (a Common Lisp implementation
of Java Debug Protocol). http://www.lichteblau.com/cloak/cl-jdi/README.html
28. Kilmer,
R., Ruby and the Java Debug Wire
Protocol: What were we thinking?, in RubyConf
2003. 2003.
29. Fowler,
C. and R. Kilmer. Ruby implementation of
the Java Debug Wire Protocol. http://rubyforge.org/projects/rubyjdwp/
30. GNU Classpath. http://www.gnu.org/software/classpath/
31. Roberts,
N., B. Rossi, and E. Zaretskii. Debugger
Machine Interface. http://www.freestandards.org/en/DMI
32. Wright,
D. and B. Freeman-Benson (2004) How to
write an Eclipse debugger. http://www.eclipse.org/articles/Article-Debugger/how-to.html
33. Kroening,
M. Amzi! Prolog: Source Code Debugger. http://www.amzi.com/manuals/amzi7/pro/pug_debugger_ide.htm
34. Winchester,
J. and A. Ryman (2001) The Pragmatics Of
Java Debugging. Sys-Con India. http://in.sys-con.com/read/36223.htm