PR python/16699: GDB Python command completion with overriden complete vs. completer class

This PR came from a Red Hat bug that was filed recently.  I checked and
it still exists on HEAD, so here's a proposed fix.  Although this is
marked as a Python backend bug, this is really about the completion
mechanism used by GDB.  Since this code reminds me of my first attempt
to make a good noodle, it took me quite some time to fix it in a
non-intrusive way.

The problem is triggered when one registers a completion method inside a
class in a Python script, rather than registering the command using a
completer class directly.  For example, consider the following script:

    class MyFirstCommand(gdb.Command):
          def __init__(self):
              gdb.Command.__init__(self,'myfirstcommand',gdb.COMMAND_USER,gdb.COMPLETE_FILENAME)

              def invoke(self,argument,from_tty):
                  raise gdb.GdbError('not implemented')

    class MySecondCommand(gdb.Command):
          def __init__(self):
              gdb.Command.__init__(self,'mysecondcommand',gdb.COMMAND_USER)

              def invoke(self,argument,from_tty):
                  raise gdb.GdbError('not implemented')

                  def complete(self,text,word):
                      return gdb.COMPLETE_FILENAME

    MyFirstCommand ()
    MySecondCommand ()

When one loads this into GDB and tries to complete filenames for both
myfirstcommand and mysecondcommand, she gets:

    (gdb) myfirstcommand /hom<TAB>
    (gdb) myfirstcommand /home/
                               ^
    ...
    (gdb) mysecondcommand /hom<TAB>
    (gdb) mysecondcommand /home
                                ^

(The "^" marks the final position of the cursor after the TAB).

So we see that myfirstcommand honors the COMPLETE_FILENAME class (as
specified in the command creation), but mysecondcommand does not.  After
some investigation, I found that the problem lies with the set of word
break characters that is used for each case.  The set should be the same
for both commands, but it is not.

During the process of deciding which type of completion should be used,
the code in gdb/completer.c:complete_line_internal analyses the command
that requested the completion and tries to determine the type of
completion wanted by checking which completion function will be called
(e.g., filename_completer for filenames, location_completer for
locations, etc.).

This all works fine for myfirstcommand, because immediately after the
command registration the Python backend already sets its completion
function to filename_completer (which then causes the
complete_line_internal function to choose the right set of word break
chars).  However, for mysecondcommand, this decision is postponed to
when the completer function is evaluated, and the Python backend uses an
internal completer (called cmdpy_completer).  complete_line_internal
doesn't know about this internal completer, and can't choose the right
set of word break chars in time, which then leads to a bad decision when
completing the "/hom" word.

So, after a few attempts, I decided to create another callback in
"struct cmd_list_element" that will be responsible for handling the case
when there is an unknown completer function for complete_line_internal
to work with.  So far, only the Python backend uses this callback, and
only when the user provides a completer method instead of registering
the command directly with a completer class.  I think this is the best
option because it not very intrusive (all the other commands will still
work normally), but especially because the whole completion code is so
messy that it would be hard to fix this without having to redesign
things.

I have regtested this on Fedora 18 x86_64, without regressions.  I also
included a testcase.

gdb/ChangeLog:
2014-09-03  Sergio Durigan Junior  <sergiodj@redhat.com>

	PR python/16699
	* cli/cli-decode.c (set_cmd_completer_handle_brkchars): New
	function.
	(add_cmd): Set "completer_handle_brkchars" to NULL.
	* cli/cli-decode.h (struct cmd_list_element)
	<completer_handle_brkchars>: New field.
	* command.h (completer_ftype_void): New typedef.
	(set_cmd_completer_handle_brkchars): New prototype.
	* completer.c (set_gdb_completion_word_break_characters): New
	function.
	(complete_line_internal): Call "completer_handle_brkchars"
	callback from command.
	* completer.h: Include "command.h".
	(set_gdb_completion_word_break_characters): New prototype.
	* python/py-cmd.c (cmdpy_completer_helper): New function.
	(cmdpy_completer_handle_brkchars): New function.
	(cmdpy_completer): Adjust to use cmdpy_completer_helper.
	(cmdpy_init): Set completer_handle_brkchars to
	cmdpy_completer_handle_brkchars.

gdb/testsuite/ChangeLog:
2014-09-03  Sergio Durigan Junior  <sergiodj@redhat.com>

	PR python/16699
	* gdb.python/py-completion.exp: New file.
	* gdb.python/py-completion.py: Likewise.

gdb/python/py-cmd.c

@@ -208,45 +208,163 @@ cmdpy_function (struct cmd_list_element *command, char *args, int from_tty)
   do_cleanups (cleanup);
 }
 
+/* Helper function for the Python command completers (both "pure"
+   completer and brkchar handler).  This function takes COMMAND, TEXT
+   and WORD and tries to call the Python method for completion with
+   these arguments.  It also takes HANDLE_BRKCHARS_P, an argument to
+   identify whether it is being called from the brkchar handler or
+   from the "pure" completer.  In the first case, it effectively calls
+   the Python method for completion, and records the PyObject in a
+   static variable (used as a "cache").  In the second case, it just
+   returns that variable, without actually calling the Python method
+   again.  This saves us one Python method call.
+
+   The reason for this two step dance is that we need to know the set
+   of "brkchars" to use early on, before we actually try to perform
+   the completion.  But if a Python command supplies a "complete"
+   method then we have to call that method first: it may return as its
+   result the kind of completion to perform and that will in turn
+   specify which brkchars to use.  IOW, we need the result of the
+   "complete" method before we actually perform the completion.
+
+   It is important to mention that this function is built on the
+   assumption that the calls to cmdpy_completer_handle_brkchars and
+   cmdpy_completer will be subsequent with nothing intervening.  This
+   is true for our completer mechanism.
+
+   This function returns the PyObject representing the Python method
+   call.  */
+
+static PyObject *
+cmdpy_completer_helper (struct cmd_list_element *command,
+			const char *text, const char *word,
+			int handle_brkchars_p)
+{
+  cmdpy_object *obj = (cmdpy_object *) get_cmd_context (command);
+  PyObject *textobj, *wordobj;
+  /* This static variable will server as a "cache" for us, in order to
+     store the PyObject that results from calling the Python
+     function.  */
+  static PyObject *resultobj = NULL;
+
+  if (handle_brkchars_p)
+    {
+      /* If we were called to handle brkchars, it means this is the
+	 first function call of two that will happen in a row.
+	 Therefore, we need to call the completer ourselves, and cache
+	 the return value in the static variable RESULTOBJ.  Then, in
+	 the second call, we can just use the value of RESULTOBJ to do
+	 our job.  */
+      if (resultobj != NULL)
+	Py_DECREF (resultobj);
+
+      resultobj = NULL;
+      if (obj == NULL)
+	error (_("Invalid invocation of Python command object."));
+      if (!PyObject_HasAttr ((PyObject *) obj, complete_cst))
+	{
+	  /* If there is no complete method, don't error.  */
+	  return NULL;
+	}
+
+      textobj = PyUnicode_Decode (text, strlen (text), host_charset (), NULL);
+      if (textobj == NULL)
+	error (_("Could not convert argument to Python string."));
+      wordobj = PyUnicode_Decode (word, sizeof (word), host_charset (), NULL);
+      if (wordobj == NULL)
+	{
+	  Py_DECREF (textobj);
+	  error (_("Could not convert argument to Python string."));
+	}
+
+      resultobj = PyObject_CallMethodObjArgs ((PyObject *) obj, complete_cst,
+					      textobj, wordobj, NULL);
+      Py_DECREF (textobj);
+      Py_DECREF (wordobj);
+      if (!resultobj)
+	{
+	  /* Just swallow errors here.  */
+	  PyErr_Clear ();
+	}
+
+      Py_XINCREF (resultobj);
+    }
+
+  return resultobj;
+}
+
+/* Python function called to determine the break characters of a
+   certain completer.  We are only interested in knowing if the
+   completer registered by the user will return one of the integer
+   codes (see COMPLETER_* symbols).  */
+
+static void
+cmdpy_completer_handle_brkchars (struct cmd_list_element *command,
+				 const char *text, const char *word)
+{
+  PyObject *resultobj = NULL;
+  struct cleanup *cleanup;
+
+  cleanup = ensure_python_env (get_current_arch (), current_language);
+
+  /* Calling our helper to obtain the PyObject of the Python
+     function.  */
+  resultobj = cmdpy_completer_helper (command, text, word, 1);
+
+  /* Check if there was an error.  */
+  if (resultobj == NULL)
+    goto done;
+
+  if (PyInt_Check (resultobj))
+    {
+      /* User code may also return one of the completion constants,
+	 thus requesting that sort of completion.  We are only
+	 interested in this kind of return.  */
+      long value;
+
+      if (!gdb_py_int_as_long (resultobj, &value))
+	{
+	  /* Ignore.  */
+	  PyErr_Clear ();
+	}
+      else if (value >= 0 && value < (long) N_COMPLETERS)
+	{
+	  /* This is the core of this function.  Depending on which
+	     completer type the Python function returns, we have to
+	     adjust the break characters accordingly.  */
+	  set_gdb_completion_word_break_characters
+	    (completers[value].completer);
+	}
+    }
+
+ done:
+
+  /* We do not call Py_XDECREF here because RESULTOBJ will be used in
+     the subsequent call to cmdpy_completer function.  */
+  do_cleanups (cleanup);
+}
+
 /* Called by gdb for command completion.  */
 
 static VEC (char_ptr) *
 cmdpy_completer (struct cmd_list_element *command,
 		 const char *text, const char *word)
 {
-  cmdpy_object *obj = (cmdpy_object *) get_cmd_context (command);
-  PyObject *textobj, *wordobj, *resultobj = NULL;
+  PyObject *resultobj = NULL;
   VEC (char_ptr) *result = NULL;
   struct cleanup *cleanup;
 
   cleanup = ensure_python_env (get_current_arch (), current_language);
 
-  if (! obj)
-    error (_("Invalid invocation of Python command object."));
-  if (! PyObject_HasAttr ((PyObject *) obj, complete_cst))
-    {
-      /* If there is no complete method, don't error -- instead, just
-	 say that there are no completions.  */
-      goto done;
-    }
+  /* Calling our helper to obtain the PyObject of the Python
+     function.  */
+  resultobj = cmdpy_completer_helper (command, text, word, 0);
 
-  textobj = PyUnicode_Decode (text, strlen (text), host_charset (), NULL);
-  if (! textobj)
-    error (_("Could not convert argument to Python string."));
-  wordobj = PyUnicode_Decode (word, strlen (word), host_charset (), NULL);
-  if (! wordobj)
-    error (_("Could not convert argument to Python string."));
-
-  resultobj = PyObject_CallMethodObjArgs ((PyObject *) obj, complete_cst,
-					  textobj, wordobj, NULL);
-  Py_DECREF (textobj);
-  Py_DECREF (wordobj);
-  if (! resultobj)
-    {
-      /* Just swallow errors here.  */
-      PyErr_Clear ();
-      goto done;
-    }
+  /* If the result object of calling the Python function is NULL, it
+     means that there was an error.  In this case, just give up and
+     return NULL.  */
+  if (resultobj == NULL)
+    goto done;
 
   result = NULL;
   if (PyInt_Check (resultobj))
@@ -302,7 +420,6 @@ cmdpy_completer (struct cmd_list_element *command,
 
  done:
 
-  Py_XDECREF (resultobj);
   do_cleanups (cleanup);
 
   return result;
@@ -548,6 +665,9 @@ cmdpy_init (PyObject *self, PyObject *args, PyObject *kw)
       set_cmd_context (cmd, self);
       set_cmd_completer (cmd, ((completetype == -1) ? cmdpy_completer
 			       : completers[completetype].completer));
+      if (completetype == -1)
+	set_cmd_completer_handle_brkchars (cmd,
+					   cmdpy_completer_handle_brkchars);
     }
   if (except.reason < 0)
     {