If you want to troubleshoot the SMTP connection to your outgoing mail server, you can do that by setting the existing debug option in the PHPMailer library, which executes the mail functions in Joomla.  This allows you to log the data, mail commands and the connection status to the PHP error log.

Please note that core modifications are not recommended and the following instructions assume that you revert the file to the original version as soon as possible.

These instruction are for Joomla 3.4.8.  If you already run Joomla 3.5.0, follow these instructions instead:  Debugging SMTP Mail in Joomla 3.5.0

Edit the file libraries/joomla/mail/mail.php and add the following lines to the beginning of the function useSmtp(), after line 467:

      // debug SMTPmailer
      // $this->SMTPDebug    = 3;        // output data, commands and connection status
      $this->SMTPDebug    = 4;           // low level data output, all messages
      $this->Debugoutput   = 'error_log';   // output to error log as configured in php.ini

In Joomla 3.4.8 you need to modify also the file libraries/vendor/phpmailer/phpmailer/class.smtp.php and replace line 190 with these lines:

        // if (is_callable($this->Debugoutput)) {
      // Avoid clash with built-in function names
        if (!in_array($this->Debugoutput, array('error_log', 'html', 'echo')) and is_callable($this->Debugoutput)) {

Once you run the mail function, for example by using Mass Mail to a small group of test users including yourself, you can see detailed messages in the PHP error log, for example this failed connection:

[19-Mar-2016 10:47:39 Europe/London] Connection: opening to smtp.gmail.com:587, timeout=10, options=array (  )

[19-Mar-2016 10:47:49 Europe/London]SMTP ERROR: Failed to connect to server: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (10060)

If the SMTP server responds and the connection succeeds, you can see the whole SMTP dialogue in the log file:

[23-Mar-2016 12:47:52 Europe/London] Connection: opening to smtp.gmail.com:587, timeout=300, options=array (  )

[23-Mar-2016 12:47:52 Europe/London] Connection: opened

[23-Mar-2016 12:47:52 Europe/London] SMTP -> get_lines(): $data is ""

[23-Mar-2016 12:47:52 Europe/London] SMTP -> get_lines(): $str is  "220 smtp.gmail.com ESMTP t7z282442970wjf - gsmtp

Make sure that you go back to the original version of the file mail.php before the log file starts growing unnecessarily.