There may be some issues with your server settings, otherwise, file_get_contents() should not be that slow. I would recommend using curl instead of file_get_contents(). Your download speed will improve with curl.
Try the following PHP code that uses curl to fetch the content from a URL.
$ci = curl_init();
curl_setopt($ci, CURLOPT_URL, $img_url);
curl_setopt($ci, CURLOPT_RETURNTRANSFER, 1);
$cont = curl_exec($ci);